stream#

Read and write large JSON arrays and convert semicolon-separated CSV files.

Requirements#

CSV conversion uses the system's sed, awk, and sort commands.

Install#

npm i @nodedk/stream

Usage#

Use a new output file. The writer appends to the destination.

var fs = require('node:fs/promises')
var stream = require('@nodedk/stream')

async function main() {
  var destination = 'records.json'
  await fs.writeFile(destination, '')

  var writer = stream.writeJSON(destination)
  writer.write({ name: 'Ada' })
  writer.write({ name: 'Kim' })
  await writer.end()

  var result = await stream.readJSON(destination, async function (record) {
    console.log(record.count, record.key, record.value.name)
  })
  console.log(result.count) // 2
}

main().catch(console.error)

API#

readJSON(filename, onData)#

Reads a top-level JSON array from a file. Optional onData({ count, key, value }) is awaited for each item: count starts at one, key is the zero-based array index, and value is the parsed item. Resolves to { count } after the stream completes.

writeJSON(filename)#

Returns { write, end } synchronously. Opens the destination in append mode; its parent directory must exist.

• writer.write(value): writes a JSON-serializable array item; returns undefined.
• writer.end(): writes the closing bracket and resolves when output finishes.

Write at least one item before end(). For an empty array, write [] directly. The writer does not expose a backpressure callback.

fromCSV(input, output)#

Converts a semicolon-delimited file. input is the source path; output defaults to replacing its final .csv with .json. Resolves to { count }, including the header row.

Use a header and at least one data row. Values are trimmed; numeric and boolean strings become native values, empty fields become '', and outer double quotes are removed. Quoted semicolons and multiline fields are not supported. Use simple paths without spaces or shell metacharacters.

name;count;active
Ada;12;true
Kim;3;false

var { fromCSV } = require('@nodedk/stream')

async function main() {
  var result = await fromCSV('records.csv', 'converted.json')
  console.log(result.count) // 3, including the header
}

main().catch(console.error)

Created by Vidar Eldøy