expose parsePattern and serializePattern

[samuelstroschein]

Context

Apps and plugins write the same parse and serialize pattern logic which can be abstracted with a function.

function parsePattern(value: string): Variant["pattern"] {
	const pattern: Variant["pattern"] = []

	// splits a pattern like "Hello {{name}}!" into an array of parts
	// "hello {{name}}, how are you?" -> ["hello ", "{{name}}", ", how are you?"]
	const parts = value.split(/(\{{.*?\}})/).filter((part) => part !== "")

	for (const part of parts) {
		// it's text
		if ((part.startsWith("{{") && part.endsWith("}}")) === false) {
			pattern.push({ type: "text", value: part })
		}
		// it's an expression (only supporting variables for now)
		else {
			const variableName = part.slice(1, -1)
			pattern.push({ type: "expression", arg: { type: "variable-reference", name: variableName } })
		}
	}

	return pattern
}

Proposal

Add a parsePattern and serializePattern function to the SDK to ease writing apps and plugins.

const pattern = parsePattern(
  "Hello {{username}}", { 
    variableReferencePattern: ["{{", "}}"]
  }
)

console.log(pattern)

[{ type: "text", value: "Hello "}, { type: "variable-reference", name: "username"}]

const serialized = serializePattern(pattern, {
  variableReferencePattern: ["{{", "}}"]
})

console.log(serialized)

"Hello {{username}}"

Additional information

• note that we previously thought that we need a serialized format to enable an abstracted parse/serialize pattern function. that's not true if we choose a function that take a config.

[felixhaeberle]

Cool!

1. Please account for spaces/empty chars in the variable definition scope with the regex. Additionally, it would be even safer/more robust if people can put in there whatever they want, including full sentences with special characters etc.
2. I hope to switch out those utility functions of Sherlock for this functionality, please have a look – I guess they are doing the same thing. https://github.com/opral/monorepo/blob/882b3ff9c9c23c865c49b1df0597958ec60e6038/inlang/source-code/ide-extension/src/utilities/messages/query.ts#L4
3. How does error detection, handling & propagation in case of not successfully parsed patterns look like?

[samuelstroschein]

1. Noted.
2. Yes, they are doing the same thing.

3. How does error detection, handling & propagation in case of not successfully parsed patterns look like?

Fallback to string. Users can fix the pattern themselves and apps have reduced complexity

i18next is able to reference other messages. Nice feature but no first class support from inlang. Instead of throwing, we can import the string as is. On export, i18next will reference the other message again.

const pattern = parsePattern("reuse $t(keyDeep.inner)")

console.log(pattern)

[{ type: "text", value: "reuse $t(keyDeep.inner)"} ]

[samuelstroschein]

Holding off an implementation because I am unsure how to implement parsing and serializing expressions in a generic way. Only variable references is easy but not enough.

Ideas welcome!

This is the pattern of an i18next message, for example. Splitting the pattern in parts could be done with {{ }} but how can the expression be parsed in a generalizable and easy to use API?

 "Some {{val, number(minimumFractionDigits: 2)}}"

{
  "type": "expression", 
  "arg": {
     "type": "variable-reference",
     "name": "val" 
  },
  "annotation": {
    "type": "function-reference", 
    "name": "number",
    "options": [
       { "minimumFractionDigists": "2" }
    ]
  }
}

[felixhaeberle]

The challenge relies in only passing variableReferencePattern: ["{{", "}}"]. You need a parser like interface to achieve generic functions.

Made a quick example: https://playcode.io/2020776

const i18nextConfig = {
    variablePattern: /\{\{(.*?)\}\}/g,
    functionPattern: /^(\w+)\((.*)\)$/,
    keyValueSeparator: ':', // i18next uses colons
    variableFormat: (name, format) => `{{${name}${format ? ', ' + format : ''}}}`,
    optionsFormat: (options) => {
        return options
            .map((option) => {
                const [key, value] = Object.entries(option)[0];
                return `${key}: ${value}`;
            })
            .join(', ');
    },
    formatFunction: (funcName, optionsString) => {
        return optionsString ? `${funcName}(${optionsString})` : funcName;
    },
};

[samuelstroschein]

Nice one, thanks. How can we simplify the API?

• using the generic function seems more complicated than a purpose built one https://github.com/opral/monorepo/blob/aa2b789ae08bcf80c268ac60b94cbbf409097335/inlang/source-code/plugins/i18next/src/import-export/importFiles.ts#L245-L292
• particularly regex makes things complicated. plains strings make things simpler e.g. variablePattern: ["{{", "}}" instead of variablePattern: /\{\{(.*?)\}\}/g