Skill
datafactory-tips
apply best practices for Fabric Data Factory
Published by Microsoft Updated Apr 6
Covers Data Engineering ETL Data Pipeline Microsoft Microsoft Fabric
Description
Operational tips and best practices for working with Microsoft Fabric Data Factory MCP tools. Use when executing queries, troubleshooting timeouts, creating dataflows end-to-end, or working with large datasets via the DataFactory.MCP tools.
SKILL.md
Data Factory Tips
Operational knowledge for working with Microsoft Fabric Data Factory.
Must / Prefer / Avoid
MUST DO
- Use
executeOption = "ApplyChangesIfNeeded"on first refresh of API-created dataflows - Re-add connections via
add_connection_to_dataflowaftersave_dataflow_definition(save may wipe them) - Use
list_workspacesand filter by name — there is nofind_workspacetool - Validate each connection with
execute_queryafter adding - Create a new dataflow instead of reverting a multi-source one to single-source — there is no
remove_connectiontool
PREFER
- Filter early in M queries to enable query folding (push filters to source)
- Expensive operations (sorting, aggregation) last — streaming operations (filter, select) first
execute_querywithcustomMashupDocumentto iterate on M code before saving- Standard engine for complex transforms; Fast Copy only for simple ingestion
AVOID
get_authoring_guidance— deprecated, author M directly- Sorting early in query chains (requires reading all data before returning results)
SELECT *equivalent without row limits during development — use "Keep First Rows" then remove
Symptom Triage
| Symptom | Likely Cause | Read |
|---|---|---|
DataflowNeverPublishedError | Default SkipApplyChanges on first run | destinations/dest-new-table.md |
DestinationColumnNotFound | Manual mappings for new table | destinations/dest-new-table.md |
| Credentials error on Lakehouse | Connection not bound | datafactory-connections.md (Binding) |
| FastCopy fails with transforms | Unsupported transform in fast copy | datafactory-advanced.md (Fast Copy) |
| Instant refresh fail (0-3s) | Privacy firewall or unpublished draft | sources/multi-source.md |
| Multi-source instant fail via API | Dirty dataflow or separate Lakehouse.Contents calls | sources/multi-source.md |
IsNewTarget = false fails | Direct navigation on API-created dataflow | destinations/dest-new-table.md |
| Stale connections after revert | save_dataflow_definition doesn't remove connections | datafactory-connections.md (Troubleshooting) |
Knowledge Files
| File | When to Read |
|---|---|
datafactory-core.md | MCP tools, M basics, rolling dates, what-if queries |
datafactory-connections.md | Connection discovery, creation, binding, gateways, troubleshooting connection errors |
datafactory-performance.md | Query timeouts, chunking, query folding, connector selection |
datafactory-advanced.md | Fast Copy limits, Action.Sequence, Modern Evaluator |
datafactory-pipelines.md | Pipeline creation, Dataflow activities, chaining, scheduling |
Destination Files (read only the one you need)
| File | When to Read |
|---|---|
destinations/dest-new-table.md | Creating a new output table via MCP (most common path) |
destinations/dest-existing-table.md | Writing to a table that already exists |
destinations/dest-troubleshooting.md | Diagnosing refresh failures, connection issues, silent errors |
Source Files
| File | When to Read |
|---|---|
sources/multi-source.md | Combining Lakehouse + SharePoint/Web sources, AllowCombine |
sources/sharepoint-excel.md | Reading Excel files from SharePoint via Web.Contents |
M Templates (copy-paste-ready, no prose)
| File | When to Read |
|---|---|
templates/m-new-table-destination.m | Need the complete M section document for a new table |
templates/m-existing-table-destination.m | Need the M section document for an existing table |
templates/m-multi-source-section.m | Need the M section document for multi-source with AllowCombine |
templates/m-sharepoint-excel-source.m | Need the M snippet for SharePoint Excel via Web.Contents |
templates/pipeline-single-dataflow.json | Need pipeline JSON for a single Dataflow activity |
templates/pipeline-chained-dataflows.json | Need pipeline JSON for chained Dataflow activities |
More skills from the DataFactory.MCP repository
View all 2 skillsMore from Microsoft
View publisherSkillTagsUpdated
rushstack-best-practices
manage Rush monorepos with best practices
rushstack
Apr 6EngineeringLocal DevelopmentMicrosoftProject Management +1azure-ai-agents-persistent-dotnet
build AI agents with Azure .NET SDK
skills
Jul 3.NETAgentsAzureLLMazure-ai-anomalydetector-java
build anomaly detection applications with Java
skills
May 13AnalyticsAzureData AnalysisJava +2azure-ai-contentsafety-java
build content moderation applications with Azure AI
skills
Jul 7AI InfrastructureAzureJavaSecurityazure-ai-contentsafety-py
detect harmful content with Azure AI Content Safety
skills
Jul 18AzureComplianceLLMMicrosoft +2azure-ai-language-conversations-py
implement conversational language understanding with Python
skills
Jul 18AnalyticsAzureLLMPython