RangeError: offset is not uint

[kimfucious]

Hi there,

I'm loving music-metadata-browser. Nice work!

Every so often, though, when processing a file, it throws an error like the below for certain songs:

RangeError: offset is not uint
    at checkOffset (index.js:1165)
    at Uint8Array.readUInt32LE (index.js:1218)
    at ID3v24TagMapper.postMap (ID3v24TagMapper.js:173)
    at ID3v24TagMapper.mapGenericTag (GenericTagMapper.js:78)
    at CombinedTagMapper.mapTag (CombinedTagMapper.js:42)
    at MetadataCollector.toCommon (MetadataCollector.js:239)
    at MetadataCollector.addTag (MetadataCollector.js:102)
    at ID3v2Parser.addTag (ID3v2Parser.js:195)
    at ID3v2Parser.parseId3Data (ID3v2Parser.js:189)

When I say "certain songs" I mean mp3 files, and that the error is consistent, in that it will always happen for some files and it will never happen on others.

What are some things to look at with regards to the file to determine why a problematic file is causing the error?

I've read through issue #15 but couldn't quite figure out how to troubleshoot the issue.

I dropped some of the files in question on the Audio Tag Analyzer, and compared those to "known good" files, but I'm not seeing thing that sticks out.

Any advice would be greatly appreciated.

P.S. I'm using "music-metadata-browser": "^1.1.1" in a create-react-app

[Borewit]

I am happy to hear you like it.

Under the assumption you suffering from the same issue as described in #15, there is nothing wrong with these 'problematic files'. These files just trigger some functionality in your buffer module which is missing.

Which is caused by an older version of buffer getting compiled / packed in your react web app.

Note that music-metadata is designed to operate node.js API. The buffer module is part of node.js. In order to get this working in the browser you need to 'shim' this functionality with a buffer replacing the standard node.js module.

If you manage to get the right version of buffer packed in, it will solve your issue.

[kimfucious]

Thanks for the tips @Borewit

My first thought was to move this process out to a serverless function and use Node version of music-metadata there, but then I realized I'd have to send the entire file to the function before it got processed.

So I took a look at your implementation of the Audio Tag Analyzer, as it seems to do this all in the browser, to see how you implemented the buffer shim.

From what I can tell, you import buffer from npm and then attach it to the window object as shown here on lines 42-43, like so:

(window as any).global = window;
import * as _buffer from 'buffer';
(window as any).Buffer = _buffer.Buffer; // note: the trailing slash is important!

I'm not using TypeScript, so my block looks like this:

window.global = window;
import * as _buffer from 'buffer';
window.Buffer = _buffer.Buffer; // note: the trailing slash is important!

Side note: I don't see a trailing slash there, but I think that's a left-over comment from ferros's docs where he's using require instead of import.

Anyhow, my attempts a "shimming" have failed, as the above doesn't seem to fix the offset is not uint error on certain "problematic" files.

I'm admittedly a toddler at the keyboard, but this stuff interests me.

If you have the time and patience, kindly let me know if I'm missing a step somewhere or if I've totally lost the plot.

Danke je,

Kim

[kimfucious]

One thing I've just noticed is that, when viewing the data on one "problematic file" in the Audio Tag Analyzer is that the file the following Common tag, where I do not see that in files that don't throw and error:

Average gain level. | 10107 | averageLevel

[Borewit]

it's not a coding issue, it's more tweaking the packager in such a way that it includes the right buffer module.

Average gain level. | 10107 | averageLevel

To decode that tag a function of the buffer module is called, which is has an in issue in the buffer module you are using. It don't think this is the only decoding part where this error can occure.

To trick the packager, you could:

• add buffer v5.2.1 as dependency in your project (npm install buffer)

Add something like this at the beginning of your code:

const BrowserBuffer= require('buffer/').Buffer  // note: the trailing slash is important!

// Trigger the buffer to ensure the packeger doesn't kick it out because it is not used
const buffer = BrowserBuffer.from([0x01, 0x02, 0x03, 0x04,]);

// Ensure the readUInt32LE is working
if (buffer.readUInt32LE() !== 0x04030201)
  throw new Error(`Expected ${buffer.readUInt32LE()} == ${0x04030201}`);

Note that I have not tested this code snippet.

Maybe you can publish your code so I can have a look.

[kimfucious]

@Borewit you're a ⭐️ !

Here's my measly attempt at coding:

import { cleanThing, uncleanThing } from "./helpers";
import { withAuthenticator } from "aws-amplify-react";
import * as _buffer from "buffer";
import * as mm from "music-metadata-browser";
import $ from "jquery";
import Amplify, { API, Storage, Auth } from "aws-amplify";
import awsconfig from "./aws-exports";
import Dropzone from "react-dropzone";
import InfoModal from "./InfoModal";
import NavBar from "./NavBar";
import React, { useEffect, useState } from "react";
import slug from "./images/slug.png";
import slugfeeding from "./images/slug_feeding.gif";
import SlugModal from "./SlugModal";

window.global = window;
window.Buffer = _buffer.Buffer;
Amplify.configure(awsconfig);
Storage.configure({ level: "public" });

const App = () => {
  const [modalAction, setModalAction] = useState("");
  const [deleteStatus, setDeleteStatus] = useState("");
  const [downloadLink, setDownloadLink] = useState("");
  const [dropzoneDisabled, setDropzoneDisabled] = useState(false);
  const [error, setError] = useState("");
  const [fileName, setFileName] = useState("");
  const [likeToggle, setLikeToggle] = useState(false);
  const [songs, setSongs] = useState([]);
  const [s3FileList, setS3FileList] = useState([]);
  const [uploadProgress, setUploadProgress] = useState(0);
  const [uploadStatus, setUploadStatus] = useState(0);

  // TODO
  // const massFeedtheSlug = async acceptedFiles => {
  //   acceptedFiles.forEach(file => {
  //     prepareToFeedTheSlug([file]);
  //   });
  // };

  const openStreamingWindow = async (fileName, index) => {
    console.log(fileName);
    const url = await Storage.get(fileName, {
      contentDisposition: "inline",
      contentType: "audio/mpeg"
    });
    window.open(url, "_blank");
  };

  let suffix;
  const prepareToFeedTheSlug = async acceptedFiles => {
    if (acceptedFiles.length > 1) {
      alert("There can be only one Highlander!");
      return;
    }
    const file = acceptedFiles[0];
    const fileName = cleanThing(file.name);
    let metadata;
    try {
      metadata = await mm.parseBlob(file);
      console.log("Metadata extracted!");
      console.log(metadata);
      let { album, title, artist, year: releaseYear } = metadata.common;
      if (title) {
        title = cleanThing(title);
      } else {
        title = cleanThing(fileName.slice(0, -4));
      }
      if (artist) {
        artist = cleanThing(artist);
      } else {
        artist = "n/a";
      }
      if (album) {
        album = cleanThing(album);
      } else {
        album = "n/a";
      }
      suffix = fileName.slice(-3).toLowerCase();
      if (suffix === "mp3" || suffix === "m4a") {
        console.log(`Ooh, a lovely ${suffix} to slurp!`);
      } else {
        alert("The slug only eats mp3s and m4as!");
        return;
      }

      console.log("Calling the Slug...");

      // check if song exists in dynamoDB, if not feed the Slug
      try {
        const response = await API.get(
          "SlugBucketApi",
          `/songs/object/${artist}/${title}`
        );
        if (
          Object.entries(response).length === 0 &&
          response.constructor === Object
        ) {
          feedTheSlug(album, artist, file, fileName, releaseYear, title);
        } else {
          const message = `The Slug has previously slurped ${response.title
            .replace(/</g, "(")
            .replace(/>/g, ")")} by ${response.artist}.`;
          console.log(message);
          alert(message);
        }
      } catch (e) {
        console.log(e);
      }
    } catch (e) {
      setError(e.message);
      console.log(e);
    }
  };

  const feedTheSlug = async (
    album,
    artist,
    file,
    fileName,
    releaseYear,
    title
  ) => {
    console.log(`Feeding the slug... ${artist} / ${title}`);
    setUploadProgress(1);
    setUploadStatus("Warming up...");
    setDropzoneDisabled(true);
    try {
      await Storage.put(`${artist} ||| ${title}.${suffix}`, file, {
        contentType: "audio/mpeg",
        // contentDisposition: "attachment",
        progressCallback(progress) {
          setUploadProgress(
            `${parseInt(100 * (progress.loaded / progress.total))}`
          );
          setUploadStatus(
            `${parseInt(100 * (progress.loaded / progress.total))}%`
          );
        }
      });
      console.log("The Slug has been fed");
      UpdateSlugBucketMetaData(artist, title, album, releaseYear, fileName);
      setTimeout(() => {
        setUploadStatus("Upload complete!");
        setDropzoneDisabled(false);
        setUploadStatus(0);
        setUploadProgress(0);
      }, 2000);
    } catch (e) {
      console.log(e);
    }
  };

  const UpdateSlugBucketMetaData = async (
    artist,
    title,
    album,
    releaseYear,
    fileName
  ) => {
    try {
      const response = await API.post("SlugBucketApi", `/songs/`, {
        body: {
          artist,
          title,
          album,
          releaseYear,
          fileName
        }
      });
      console.log("Slug Bucket Metadata Updated!");
      console.log(response);
    } catch (e) {
      console.log(e);
    }
  };
  const getUrl = async (fileName, action) => {
    console.log("geturl filename", fileName);
    try {
      const url = await Storage.get(fileName);
      setFileName(fileName);
      setDownloadLink(url);
      setModalAction(action);
    } catch (e) {
      console.log(e);
    }
  };

  const handleDelete = file => {
    console.log("file name to delete", file);
    Storage.remove(file)
      .then(result => {
        console.log(result);
        setDeleteStatus("deleted");
        setFileName("");
        $("#slug-modal").modal("hide");
      })
      .catch(err => console.log(err));
  };

  const handleLike = async (artist, title, fileName) => {
    const who = await Auth.currentAuthenticatedUser();
    console.log(who.username);
    const songsTemp = songs;
    songsTemp.map(song => {
      if (song.fileName === fileName) {
        console.log("Match!");
        if (song.liked) {
          delete song.liked;
        } else {
          song.liked = true;
        }
        console.log(`${song.liked ? "Liking" : "Unliking"} song on DynamoDB`);
        try {
          API.put("SlugBucketApi", "/songs", {
            body: { ...song }
          });
          console.log(
            `${title} by ${artist} has been ${song.liked ? "liked" : "unliked"}`
          );
        } catch (e) {
          console.log(e);
        }
      }
      return song;
    });
    await setSongs(songsTemp);
    setLikeToggle(!likeToggle);
  };

  useEffect(() => {
    console.log("Event: Like/Unlike");
  }, [likeToggle]);

  const renderFileList = () => {
    if (s3FileList.length === 0) {
      return (
        <div className="lead text-center">
          The slug bucket is empty (or loading)...
        </div>
      );
    } else {
      console.log("Slurping...");
      return songs.map(file => {
        if (!file.album) file.album = "n/a";
        const index = songs.indexOf(file);
        const { album, artist, title, fileName, liked } = file;
        const slugBucketFileName = `${artist} ||| ${title}.${fileName.slice(
          -3
        )}`;
        return (
          <li
            key={index}
            className="list-group-item d-flex flex-column justify-contents-center"
          >
            <div className="d-flex justify-content-between w-100">
              <div className="d-flex flex-column">
                <div className="d-flex align-items-center">
                  <i
                    className="material-icons mr-2 text-secondary"
                    id={`playSong-${index}`}
                    onClick={() =>
                      openStreamingWindow(slugBucketFileName, index)
                    }
                    style={{ cursor: "pointer" }}
                  >
                    play_circle_filled
                  </i>
                  <div className="lead mr-auto">{uncleanThing(title)}</div>
                </div>
                <div>
                  <small>
                    <strong>Artist: </strong>
                    {uncleanThing(artist)}&nbsp;&nbsp;
                    <strong>Album: </strong>
                    {uncleanThing(album)}
                  </small>
                </div>
              </div>
              <div className="d-flex align-items-center">
                <i
                  onClick={() => handleLike(artist, title, fileName)}
                  className={` material-icons mx-1 ${
                    liked ? "text-danger" : "text-muted"
                  }`}
                  style={{ cursor: "pointer" }}
                >
                  {liked ? "favorite" : "favorite_border"}
                </i>
                <i
                  className="material-icons mx-1 text-primary"
                  onClick={() => getUrl(slugBucketFileName, "download")}
                  style={{ cursor: "pointer" }}
                  data-toggle="modal"
                  data-target="#slug-modal"
                >
                  cloud_download
                </i>
                <i
                  onClick={() => {
                    setFileName(slugBucketFileName);
                    setModalAction("delete");
                  }}
                  className="material-icons mx-1 text-danger"
                  style={{ cursor: "pointer" }}
                  data-toggle="modal"
                  data-target="#slug-modal"
                >
                  delete_forever
                </i>
              </div>
            </div>
          </li>
        );
      });
    }
  };

  const getS3Files = async () => {
    try {
      const response = await Storage.list("", { level: "public" });
      setS3FileList(response);
    } catch (e) {
      console.log(e);
    }
  };

  useEffect(() => {
    console.log("Effect! Getting...");
    getS3Files();
  }, []);

  useEffect(() => {
    console.log("Effect! Getting metadata...");
    const getSong = async song => {
      const artist = song.key.split(" ||| ")[0];
      const title = song.key.split(" ||| ")[1].slice(0, -4);
      let metadata = await API.get(
        "SlugBucketApi",
        `/songs/object/${artist}/${title}`
      );
      if (
        Object.entries(metadata).length === 0 &&
        metadata.constructor === Object
      ) {
        metadata = { artist, title, fileName: song.key };
        console.log("Dynamo metadata is missing ! Trying to add...");
        try {
          await API.post("SlugBucketApi", "/songs", {
            body: {
              artist,
              title,
              fileName: song.key
            }
          });
          console.log("Song metadata updated!");
        } catch (e) {
          console.log(e);
        }
      }
      console.log("Here's your metadata", metadata);
      return metadata;
    };
    const pending = s3FileList.map(song => getSong(song));
    Promise.all(pending).then(songs => setSongs(songs));
  }, [s3FileList]);

  useEffect(() => {
    console.log(
      `Effect | Upload Status: ${uploadStatus}, DeleteStatus: ${deleteStatus}`
    );
    if (uploadStatus === "Upload complete!" || deleteStatus === "deleted") {
      getS3Files();
      setDeleteStatus("");
      setUploadStatus(0);
    }
  }, [deleteStatus, uploadProgress, uploadStatus]);

  return (
    <>
      <NavBar />
      <div
        className="d-flex flex-column justify-content-center align-items-center container mt-5"
        // style={{ position: "relative" }}
      >
        {error ? (
          <div
            className="alert alert-dismissible alert-danger"
            style={{
              position: "absolute",
              bottom: "0px",
              zIndex: "999"
            }}
          >
            <button
              type="button"
              className="close"
              onClick={() => setError("")}
              style={{ "&:hover": { opacity: "1" } }}
            >
              &times;
            </button>
            <strong>Oh, nos! The Slug has indigestion.</strong>
            <div>{`Message for Slug:  "${error}"`}</div>
          </div>
        ) : null}
        <div className="d-block d-sm-none display-4 text-center">
          Slug Bucket
        </div>
        <div className="d-none d-sm-block display-2 text-center">
          Slug Bucket
        </div>
        <div className="d-flex justify-content-center input-group mb-3 text-center">
          <Dropzone
            onDrop={acceptedFiles => {
              // massFeedTheSlug(acceptedFiles);
              prepareToFeedTheSlug(acceptedFiles);
            }}
            disabled={dropzoneDisabled}
          >
            {({ getRootProps, getInputProps }) => (
              <section>
                <div style={{ outline: "none" }} {...getRootProps()}>
                  <input {...getInputProps()} />
                  <img
                    className="m-3"
                    src={dropzoneDisabled ? slugfeeding : slug}
                    alt="slug bucket logo"
                    height="201"
                    width="200px"
                    title="Feed me!"
                    style={{ cursor: "pointer" }}
                  />
                  <p className="lead">
                    {uploadProgress > 0 && uploadProgress < 100
                      ? "The Slug is Feeding!"
                      : "Feed the Slug!"}
                  </p>
                </div>
              </section>
            )}
          </Dropzone>
        </div>
        {uploadProgress > 0 ? (
          <div className="d-flex flex-column align-items-center justify-content-center w-100">
            <div
              className="progress"
              style={{ maxWidth: "768px", minWidth: "300px" }}
            >
              <div
                className={`progress-bar progress-bar-striped ${
                  uploadProgress < 100
                    ? "progress-bar-animated bg-primary"
                    : "bg-success"
                }`}
                role="progressbar"
                aria-valuenow={uploadProgress}
                aria-valuemin="0"
                aria-valuemax="100"
                style={{ width: `${uploadProgress}%` }}
              />
            </div>
            <div className="text-center lead text-muted mb-3">
              <small>{uploadStatus ? uploadStatus : null}</small>
            </div>
          </div>
        ) : null}
        {songs.length > 0 ? (
          <>
            <div
              className="align-items-center d-flex d-sm-none display-4 mb-3"
              style={{ fontSize: "24px" }}
            >
              <span>Songs in the Key of Slug</span>
              <i
                className="material-icons text-muted ml-2"
                data-toggle="modal"
                data-target="#info-modal"
                title="How to Slug..."
                style={{ cursor: "pointer" }}
              >
                info_outline
              </i>
            </div>
            <div
              className="align-items-center d-none d-sm-flex display-4 mb-3"
              style={{ fontSize: "32px" }}
            >
              <span>Songs in the Key of Slug</span>
              <i
                className="material-icons text-muted ml-2"
                data-toggle="modal"
                data-target="#info-modal"
                title="How to Slug..."
                style={{ cursor: "pointer" }}
              >
                info_outline
              </i>
            </div>
          </>
        ) : null}
        <ul
          className="d-flex justify-content-center list-group w-100 mb-5"
          style={{ maxWidth: "800px", minWidth: "300px" }}
        >
          {renderFileList()}
        </ul>
        <SlugModal
          action={modalAction}
          artist={fileName.split(" ||| ")[0]}
          title={fileName.slice(0, -4).split(" ||| ")[1]}
          fileName={fileName}
          url={downloadLink}
          fn={handleDelete}
        />
        <InfoModal />
      </div>
    </>
  );
};

export default withAuthenticator(App, true);

[kimfucious]

More than you wanted, or needed, I'm sure 😸

[Borewit]

Can you please share the entire project in a repo?

Ideally it is project setting to get the right buffer in.

[kimfucious]

Sure, thanks @Borewit

Let me check that repo before I make it public, as it's part of a CI deploy via Amplify, and I'm not sure how much private info regarding my AWS resources is exposed, though I'm pretty sure those resources are locked down regardless.

[Borewit]

Make sure nothing sensitive is in your history neither.

You can consider granting me access to the private repo if you fear going public. Or maybe copy all files required and push a to new repo.

[kimfucious]

Cool; thanks. Almost there.

[Borewit]

You make me curious.

[Borewit]

Any luck Kim?

[kimfucious]

Ha ha! Sorry for the delay. And thanks for the follow up.

I'm battling with getting a "clean" repo up. Amplify CLI uses a key file (with all my goodies in there) for the build. If I remove that file, then the build fails.

I've just learned of another way to do this, and will let you know if/when I publicize my repo.

[dansharisan]

Any updates yet please? I'm also having this issue and is struggling to solve it.

[Borewit]

How can I reproduce the issue @dansharisan? Would a react app be sufficient?