Data sources

A data source keeps your catalog in sync with a file that is kept somewhere else, so when that file changes your catalog follows without you opening it.

A data source is a supplier’s spreadsheet that the PIM fetches and reads into your catalog over and over, on a schedule you choose. You tell it where the file is, which column feeds which field, and how often to look. After that it runs on its own, without you.

You would set one up for one of three reasons. A supplier whose prices and stock change on you without warning. Several suppliers sending you the same product, with no agreement about whose number to keep. Or a file you re-upload every week and do not want to map again each time.

A data source only ever updates records you already have. It matches each row of your file to something in your catalog and writes into it. A row that matches nothing is skipped instead of becoming a new product, so loading a catalog for the first time is an import.

You will find all of this under Uploads, in the Data Sources tab. The Create data source button above the table opens a dialog over the page with up to 7 screens in it. The table under it lists every source your team has built, with a health badge on each one.

Before you start

Whether you want an import instead

Both read a CSV or an XLSX file into your catalog, and that is where the resemblance stops. An import is a file you run once. A data source is a file the PIM keeps going back to for you.

Choose an import for a one-off load, and for anything that has to create products, because a data source never creates one. Choose a data source when a supplier’s numbers move on their own. Your catalog then follows them without anyone opening the file.

What your role lets you do

Your Data Sources tab is Manager and above. An Admin and a Manager see it and can do everything on it: create a source, run it, pause it, edit it and delete it.

An Editor does not see the tab at all. Someone on that role who follows a link to it gets Imports instead. You set roles in team, roles and permissions.

A blank cell clears what you had

This is the behavior that surprises people arriving from imports, where a blank cell leaves your value alone. What a sync writes to your catalog, further down, says which fields clear and which are protected.

The records have to exist first

A row is only useful once it finds something in your catalog to write into. On an empty catalog every row of your file is skipped, so load the catalog with an import first.

Connecting a feed too early is not wasted work. Rows that found nothing get re-checked by themselves as soon as your catalog gains something they could match. An import you run is what usually sets that off.

How to set up a data source

1. 1

   Open the dialog and name the source

   From Uploads, select the Data Sources tab and click Create data source above the table. On the first screen, called Basics, type a Name. Then set the Entity dropdown to the kind of record your file is about.

2. 2

   Point it at the file

   On Source, set the Source Type dropdown to where your file is, then fill in the address or drop the file in. Clicking Next connects to the server and reads your file’s columns.

3. 3

   Say how a row finds its record

   On Rules, each pair of dropdowns is one kind of identity on the left, such as SKU, and the column of your file with it in it on the right. The PIM has already guessed both from your file.

4. 4

   Say which column feeds which field

   On Fields, every catalog field is a row with a dropdown of your file’s columns. The PIM has matched the ones whose names line up with your headers. Clear any you do not want fed.

5. 5

   Set the options

   Options appears only when something you mapped needs a decision. Every section on it has a default, so you can click Next without opening one.

6. 6

   Match the file's brand names to yours

   Brands appears only on a Products source that matches on Brand Name. Point each brand name in your file at a brand in your catalog, or click Next and skip it.

7. 7

   Choose how often it runs

   On Schedule, set the Frequency dropdown to Interval, Daily or Weekly, fill in the times, and click Create.

That is the whole flow, and you are done. Your source appears in the table right away. It runs for the first time when its schedule comes around, or as soon as you click Run now in its row.

Everything below explains those screens one at a time, then what a sync does to your catalog. You do not need any of it for a straightforward feed.

Every option in detail

The dialog has 7 screens and you see between 4 and 7 of them. Options appears only when something you mapped brings a decision with it. Brands appears only on a Products source whose rule uses Brand Name, and Schedule never appears when you upload the file yourself.

Click Back at any point, or click a step marker along the top of the dialog to jump to a screen you have already filled in. Next stays gray until the screen you are on is complete. On the last screen that button reads Create, or Save when you are editing a source that already exists.

Step 1: Name it and choose what it feeds

Basics is the first screen inside the dialog, and you fill in three fields.

Both text boxes count your characters under them as you type.

Entity is the choice everything else you do hangs on. It decides which identities you can match on, which fields you can feed, and which options appear later. One source feeds exactly one kind of record, so a file with products and brands together needs two sources.

Step 2: Point it at the file

Source is the second screen, and it asks you where your file is. Source Type is a dropdown with 4 choices, and the fields under it change with your answer.

Port fills itself in when you pick a protocol, 21 for FTP and 22 for SFTP. A port you typed yourself is kept when you switch between the two.

On FTP the username and the password go together. Leave both Username and Password empty for a server that accepts anonymous access. Fill both in otherwise, because typing one makes the other required. SFTP always wants both from you. The eye button beside Password shows you what you typed.

Clicking Next on this screen actually connects. The PIM connects to the server, downloads your file, and reads its column names and its first rows. Those columns are what the screens after this one give you.

A failure keeps you on this screen with a red banner naming the reason, so you can fix it and click Next again.

Step 2: Upload the file yourself

Manual Upload gives you a drop area instead of the address fields. Drag your file onto it, or click it to browse. The PIM reads your file as soon as it arrives and shows you its column count, a few sample rows and its total. Drop a new file on the same area at any time to replace what is there.

An XLSX with more than one sheet gets a Sheet picker above that preview. Sheets the PIM can see are empty stay out of your picker. When the first sheet is one of those, it selects a sheet with data for you.

A file you upload has no schedule. It syncs the moment you save the source. After that it runs only when you upload a new file or click Run now on its row, and editing a field mapping on its own does not run it.

Step 3: Say how a row finds its record

Rules is the third screen, and it is the one that does the matching for you. It is where you say what makes a row of your file the same thing as a record in your catalog.

A rule is a list of pairs you build. Each pair is two dropdowns side by side. On the left you pick the kind of identity, such as SKU or Barcode. On the right you pick the column of your file it is in. Rule 1 is already there when you arrive, with the column filled in wherever the PIM recognized one of your headers.

Single Match uses one pair and Compound Match uses two or more. The two are radio buttons at the top of each rule. A single match takes one of your columns and looks for the record with that exact value. A compound match takes several of your columns and looks for the one record that satisfies all of them at once.

That is why Brand Name and Product Type appear inside a compound rule and nowhere else. Either one alone would match every product of that brand or that type.

Rules are tried from the top down, and the first one that finds exactly one record is the one used. Drag a rule by the handle on its left edge to move it up or down. Add rule at the bottom of the screen adds another rule, and Add inside a rule adds another pair to that rule.

A rule that finds more than one record stops the row there. The row counts as not matched, the rules below it are never tried, and nothing is written. Writing your supplier’s price into one of two products sharing a title is worse than writing nothing.

One of your rules cannot use the same kind of identity twice, and two of them cannot be identical. The screen says so either way, with Duplicate target type within this rule. Change one of them. and Same as Rule 1 - change target or column. A compound rule with one pair left says Compound rules require at least 2 match fields. Add another field or switch to Single Match.

Step 3: What matching decides on a products feed

On a Products source, the identity you match on decides how much of each row goes in.

A per-variant match writes only what belongs to that variant when the product has more than one. You get its price, its cost, its stock, its measurements and its barcodes. The fields the whole product shares are dropped with no message, which sync history covers along with the exception images get.

A product-level match writes to the product and to its default variant, the row variants calls the hidden one. So a feed you mean to own titles and descriptions matches on Slug or Title. A feed matching on SKU is a per-variant feed whatever else is in its columns.

Step 4: Say which column feeds which field

Fields is the fourth screen, and it is where you connect the two. Every catalog field for your Entity is one row, with the field name on the left and a dropdown of your file’s columns on the right. Pick a column to feed that field, or leave the dropdown empty to leave the field alone.

The PIM fills this in for you before you touch it. When you arrive, every field whose name matches one of your headers is already mapped. Capitals, spaces and punctuation are ignored, so a column called part number goes to Part Numbers by itself. Clearing a mapping sticks, because that guess runs once and never puts a field back after you take it out.

The fields are sorted into groups with a heading each, in the same order the export dialog uses. Above them you get a toolbar with a Search box and a filter with All, Mapped and Unmapped. The counts beside those three stay over your whole field list while you narrow it, so they do not move as you type.

You need at least one field mapped to save. Below that, Next stays gray on this screen and Save stays gray on the last one.

Products get one row no other kind of record has, called Custom, at the bottom of the Identifiers group. Point it at any column and that column becomes a custom identifier. Its header becomes the label, and each cell becomes that product’s value. It is the one row here the PIM never guesses for you, and one source has one of them, so picking a different column replaces the first.

Step 5: Set the options

Options is the fifth screen and it has up to 4 sections. Each one is collapsed when you arrive, and opening one closes the last. A section appears only when you mapped a field that needs it, and when none of the four apply the screen is skipped.

Missing references asks one question per thing you mapped. A products feed can raise 6 of them: If Brand does not exist:, If Tag does not exist:, If Category does not exist:, If Shopify Category does not exist:, If Spec Value does not exist: and If Spec is not allowed for the product's categories:. Each one has two answers: Skip with warning, which is the default, and Fail the row. There is no third answer, because a data source does not create the missing record.

Pick Skip with warning while you are getting a feed working, so one unknown brand does not throw away the price and the stock on that row as well. Pick Fail the row once your feed is settled and you would rather see a row fail loudly than write half of it.

Asset downloads covers links that do not work. If an asset download fails: starts on Warn and continue, against Fail row. On a products feed that maps a gallery column but no primary image column, a second question appears underneath. Its two answers are Leave without primary and Promote first to primary, and the second one takes the first gallery image that actually downloaded.

If an asset already exists decides what a file you already have does to its own labels. Reuse, update labels is the default and overwrites your labels with the feed’s. Reuse, keep labels leaves your labels alone, and Upload as new stores another copy every time.

Array value separators is for cells with a list in them. A product with three tags fits them into one cell, and the PIM splits that cell on a pipe. The parts of a category path split on >. Change the character here if your own values already contain a pipe. You get one box per list the record can take, each takes up to 10 characters, and clearing a box puts the default back.

Step 6: Match the file’s brand names to yours

Brands shows up only on a Products source whose compound rule uses Brand Name. It lists every distinct brand name in your file and lets you point each one at up to 15 brands in your catalog, or mark it + Create as new brand.

A badge on each row says where it stands. Mapped means you pointed it somewhere, New means you marked it for creating, and Needed means you have not answered it yet. A search box and an All, Mapped, Unmapped filter are above the list, the same pair the Fields screen has.

This is the one place in the whole feature where a data source creates a record. A brand name is one thing or the other, never both, and brands is where the created ones appear. You can also click Next and skip the screen, which leaves those rows to your If Brand does not exist: answer.

Step 7: Choose how often it runs

Schedule is the last screen for a file the PIM fetches. Frequency is a dropdown with 3 choices, and the fields under it change with your answer.

Daily is not the same thing as every 24 hours. Daily and Weekly fire at the clock times you chose, in the timezone your schedule was saved in. Editing it later never moves it to whoever edited it. Interval counts hours from whenever the last run happened.

Choose Interval for stock, where you want the number close to the truth all day and 5 minutes is the shortest gap on offer. Choose Daily or Weekly for a supplier who publishes a fresh file at a known hour, so your run happens after the file changes.

A due source starts within about a minute of falling due. Changing your schedule takes effect immediately, so a weekly source you re-time does not wait out the old week first.

A source runs one at a time and never doubles up. When a slot comes around while the last run is still going, that slot is skipped and the next minute tries again. A stretch of downtime gives you one run and then the normal rhythm, never a pile of catch-up runs at once.

What the wizard does not let you change

Some things about reading your file are decided for you.

Run, pause or delete a source

Everything you do to a source after it exists happens on its row in the list. Clicking the row itself reopens the wizard on that source. Saving an edit is all or nothing: the source, the credentials, the rules, the mappings and the brand mappings are stored together or none of them are.

Check the boxes on several rows and a bar appears above the table. You get Run Now, Pause / Resume and Delete on it, doing the same three things to all of them at once.

Run now is more than running early. A scheduled run leaves alone the rows that were sealed as not matched and the rows whose errors have settled. Run now puts all of those back in line, which is how you apply a fix you made yourself. You do not have to wait for your supplier to touch that row again.

The button is inactive while the source is paused, while a run is already going, and on an uploaded file with nothing uploaded yet. It comes back on its own when the run finishes.

Pause stops a feed without erasing it. Its schedule skips a paused source, and so does Run now. It also stops competing for the fields it maps, so a source you pause does not hand its fields to another one the next day. Resume puts it back on its normal schedule.

Every value that source wrote stays in your catalog exactly as it left it, and the sources that remain re-decide the fields they also map. The row leaves your list at once and its name is free to use again immediately, while its history is cleared out in the background.

What the list shows you

The table has 15 columns. Name, Description, Source, Target, Match Rules, Mapped Fields, Frequency, Status, Last Sync, Matched, Unmatched, Created at, Created by, Updated at and Updated by. Hovering Match Rules or Mapped Fields spells out what they mean.

Matched and Unmatched are the running answer to how much of that source’s file has found a record in your catalog. A source whose Unmatched climbs is a source whose file has moved on from your rules.

One more button is on the toolbar above the table, beside Create data source. Field priorities opens the screen where you say which source your catalog believes when two of them send the same field, and field priorities covers all of it.

Limits

What a data source cannot do

What a sync writes to your catalog

Every run checks every matched row

A run does not only look at what changed. Every scheduled run walks every row that has a record in your catalog. It works out what your feeds and your field priorities say that record should have, and writes it if your catalog says something else. A row nothing disagrees about is left completely alone, so a settled catalog costs you nothing and shows you nothing.

Understand that before you edit a fed field yourself, because it means a data source owns its columns for as long as it is running. Change a supplier’s price in the product editor and the next run puts the supplier’s number back. To keep your edit, stop the source from feeding that field: clear the mapping on Fields, or pause the source.

It also means a value that went wrong repairs itself. A field someone overwrote by accident comes back on the next run without you doing anything.

What a blank cell does

A blank cell is a value your supplier is asserting, not a gap they forgot. Which value depends on the field.

A column your supplier stops sending altogether is the opposite case, and it is safe. The PIM only writes fields it was given. A column that disappears from your file leaves your value alone forever, and a column that is present and empty is the one to watch.

Why a zero in a money cell counts as empty

A price, a cost, a MAP, an MSRP or a compare-at price with 0, 0.00 or $0 in it is read as no value at all. It clears the column exactly as a blank clears it, and a measurement of 0 is read the same way.

The reason is that a supplier writing 0 almost always means they have no number for you, not that the item is free. So no feed can put a literal zero into a money column, whatever your supplier sends. The one way to set one is to type it into the product’s money fields yourself, which do not go through this reading step.

Supplier Stock is the exception, where 0 is a real answer and stays one.

What happens when a row stops arriving

Nothing in your catalog is deleted when a row stops arriving. Every field that row was writing keeps the last value it wrote. A discontinued product does not vanish, does not unpublish and does not empty out.

There is one exception, and it is the one you want. When a product’s rows stop arriving from every source that feeds stock, Supplier Stock drops to 0 within seconds. Effective Stock then falls back to the on-hand and fallback numbers you own, which is what inventory means by a feed-owned column. Your catalog stops advertising a quantity no one is confirming.

What happens to a row that never matched

A row that arrived and matched nothing is sealed rather than retried forever. A supplier’s file full of products you do not carry costs you nothing every run.

The PIM re-opens those rows by itself as soon as your catalog gains something they could now match. Three things count: a product created, an identifier added or changed, and a brand renamed or attached to a product. Rewriting a title does not count, because a title is not what those rows failed on.

An import finishing sets that check off within seconds, which is what makes the normal order work. Connect your feeds, then load your catalog, and your feeds catch up on their own. For a file you upload yourself, an import finishing is the only thing that re-opens those rows apart from Run now.

When something goes wrong

Reading the badge on the list

The Status column shows one badge, and two of the words do not mean what they look like.

Error and Needs attention show you the reason on hover. The others show nothing.

When a source switches itself off

A source switches itself off after 5 failed runs in a row, and reads Needs attention from then on. Only whole-file failures count toward that 5: a download that never arrived, a file that would not open, a save that failed.

Rows that fail to apply do not count, and neither does a run where the file was fine and only some rows failed. So your source can report row errors run after run without ever being switched off, and nothing lets you change the number.

A source in Needs attention still runs when you click Run now. One successful run puts it back to Active and resets the count.

When the file’s columns change

There is one failure that switches a source off the very first time it happens: a column your rules or your mappings use disappearing from your file. Nothing is written, because a column present in your setup and absent from your file would read as an empty cell and clear the field.

Reopening the wizard puts The file's columns have changed. Fix the fields marked in red below before continuing. at the top. Every broken dropdown is marked in red, with This column is no longer in the file under each one. Pick the new column for each, and saving reactivates your source, clears the error and runs it.

When a row matches more than one record

A row whose rule found several candidates writes nothing and counts as not matched. To see why, open History on that source and download its Results report. That row’s message names up to 5 of the records it collided with, and tells you what to change.

Products are named by their slug there. In the usual case, two products with the same title, the titles are identical and the slug is the only thing telling them apart. Identifiers covers what makes a value unique in the first place.

When one file names the same product twice

Two rows of the same file pointing at the same record both get through, and only one set of values is kept. Every row of that group shows the warning in its message column of the Results file. A repeated SKU in your supplier’s own file is something you can find rather than something you guess at.

The emails you get

Every whole-file failure emails you, one message per source. It gives you the same sentence the badge shows on hover, and a button back to your list. Repeated failures collapse into one message, so a source failing every 5 minutes does not fill your inbox, and the next successful sync closes it.

Email is the only channel there is. A run where the file was fine and only some of its rows failed emails no one. A feed with rows failing without a word is one you find in sync history and not in your inbox.

Where to go next