90,419 questions
-3
votes
0
answers
68
views
CSV Flat File Source error / text truncation issue [closed]
I am loading a cleaned CSV file into SSIS using a Flat File Source, but the Data Flow keeps failing with the following error:
Data conversion failed. The data conversion for column "...
-3
votes
0
answers
47
views
CSV data not outputting from list comprehension after record count (for loop) [duplicate]
I've been tasked to manipulate a CSV file in Python. Set up a strip & split command to clean up the data, and it works:
with open("GLB.Ts+dSST_cleaned.csv") as csv:
header ...
-1
votes
0
answers
61
views
How to install league/csv bundle with Doctrine v3.3 on Symfony?
I am having a problem today installing the league/csv bundle (v9.27) on my Symfony (v7.2) project with doctrine/orm (v3.3)! When I run command php require league/csv, I get this message:
Your ...
Tooling
0
votes
7
replies
105
views
One liner to get distinct values of all columns of a tsv
I am looking for a one liner that could be run in a linux terminal that does the below.
Takes as input a tab separated file (tsv) with many columns (~100) and creates a two column tsv output with ...
0
votes
0
answers
72
views
polars.LazyFrame.sink_csv does not give CRLF line termination [duplicate]
I have a Python file
import polars as pl
import requests
from pathlib import Path
url = "https://raw.githubusercontent.com/leanhdung1994/files/main/processedStep1_enwiktionary_namespace_0_43....
1
vote
3
answers
168
views
Polars: how to write a column of strings into a txt file without escaping?
I have a .ndjson files with millions of rows. Each row has a field html which contains html strings. I would like to write all such html into a .txt file. One html is into one line of the .txt file. I ...
-3
votes
0
answers
71
views
Python - updating CSV file [closed]
I am making a To-do app and the priority,status,time left to finish and the name is saved in a CSV file.When the status is updated the the CSV file status should also be updated this is the code when ...
Advice
0
votes
2
replies
83
views
CSV file editing via bat file
I have a report CSV file that has some special characters in the header row. I would like to set up a short script in a .bat file to remove these characters, so i can schedule a task to automatically ...
1
vote
1
answer
96
views
Python, parse nested JSON to make it flat for CSV
I'm trying to store API output into CSV/db and can not figure out how I can make for those Key in "tierList". One row in my case should be on bin and I need key as a columns in my output.
Is ...
-3
votes
1
answer
97
views
create dataframe from csv in PythonAnywhere [closed]
I am trying to display the headers of a data frame I created based on a csv file using the PythonAnywhere free version. I keep getting a huge error message and I don't understand what I did wrong.
...
0
votes
1
answer
59
views
Import-Csv on file with `"#"` in the header
I'm importing a CSV generated by the chess cli tool Ordo. It has this output format:
"#","PLAYER","RATING","ERROR","POINTS","PLAYED","(%...
Best practices
0
votes
2
replies
74
views
Index timestamp shiny for python
def prepare_dataframe(df):
df.rename(columns={
'Bomba Calor - Temperatura de Aire (°C)': 'temp_aire',
'Bomba Calor - Temperatura Entrada (°C)': 'temp_entrada',
'Bomba Calor ...
0
votes
0
answers
70
views
Reading in values from CSV and making sure they are non-scientific format in R? [duplicate]
Assume I have the following Excel Sheet:
Location
Mar2000
London
1234567891011
Tokyo
12345667897
These are the raw values saved in a CSV format e.g. my_data.csv (assume it is CSV not UTF-8 format). ...
Best practices
1
vote
4
replies
121
views
Fastest way to count lines in a .csv from a .bat file?
I have this .bat file that I use to open .csv files, which counts the lines in them and uses Excel to open if under a million records and a separate program to open if more than a million records. The ...
1
vote
1
answer
81
views
Converting JSON to CSV using jq with header line
I have a task to convert a JSON file to a CSV with a header at the top. The header has to be in a particular order. So the order of the keys in the JSON file will be in different order than the ...
2
votes
0
answers
103
views
Loading a large multiline CSV file using pyspark is extremely slow
I've got a multiline CSV file which is about 150GB and I've been trying to load it using the usual code e.g.
df = spark.read.format('csv').option('header', True).option('multiLine', True).load('path/...
1
vote
1
answer
88
views
Saving Movement Data as a CSV File in VR Headset (C#/Unity)
I'm working on a project where I want to collect movement data (X/Y/Z position and rotation) from a VR headset (a PICO Neo 3 Pro Eye) using a C#/Unity script, and export it as a CSV.
I have a script ...
1
vote
0
answers
50
views
Issue with Instantiating CsvValidator.ValidatorBuilder Due to Package-Private Access in csv-validator-java-api
I've been working with the csv-validator-java-api and came across a potential discrepancy in its documentation (or maybe I misunderstood how to use it). According to the documentation, the inner ...
1
vote
3
answers
163
views
How to lazily load DataTable rows from a large text file using IEnumerable and yield return in C#?
I'm working on a C# application where I need to read a very large text file and convert its contents into a DataTable. However, I don't always need all the rows—so I want to avoid loading the entire ...
2
votes
0
answers
75
views
How do you get specific data in unknown row from a csv file using known data from the same row? [duplicate]
import geopy # used to get location
from geopy.geocoders import Nominatim
import pandas as pd
from pyproj import Transformer
def get_user_location(): # user location
geolocator = Nominatim(...
1
vote
1
answer
44
views
pandas.read_csv uses only utf-8 encoding for django file upload
I'm testing django using file uploads. Was facing a strange issue, when despide which encoding I choose, I'm always getting same error message that pandas is trying to decode with UTF-8
pd.read_csv(...
3
votes
3
answers
128
views
Converting a CSV file into a nested JSON file using ./jq
I am hoping I can get an answer on this question. I am using ./jq to convert a CSV file into a file of JSON objects. Each line of the CSV input represents a JSON object. A sample of the CSV input is ...
0
votes
0
answers
65
views
Way to transform a BalanceSheet from QBO inside BigQuery?
I'm using dbt models to transform a QBO Balance Sheet uploaded as a CSV into Bigquery.
Data comes from a fictional company inside QuickBooks test drives from which I export reports in CSV format.
Here'...
2
votes
1
answer
88
views
How to combine multiple rows of Pandas dataframe into one row using a key [duplicate]
I am trying to manipulate a CSV using Pandas and I need to get the data into the format of one row per ID.
This is an example of what I am trying to accomplish:
From:
df = pd.DataFrame({
'ID': [1, 1, ...
0
votes
0
answers
50
views
GAMS csv importing and load issue
i am very new to GAMS and im trying to understand how sets work while importing from CSV and what im doing wrong. I have data as in picture below (prices_wide_example.csv), i would like to import it ...
0
votes
0
answers
48
views
Databricks COPY INTO from CSV file not returning any rows
I've seen a lot of other people having issues with the COPY INTO command on Databricks, but I haven't found any answers for this exact problem yet.
I'm working on learning Databricks sql syntax by ...
0
votes
0
answers
35
views
Databricks group cluster fails to read CSV (TextFileFormatEdge$.disabled) while personal cluster works
I have a PySpark function that reads a reference CSV file inside a larger ETL pipeline.
On my personal Databricks cluster, this works fine. On the group cluster, it return empty dataframe, the same ...
0
votes
1
answer
96
views
How to format a vocabulary list into a table using Python for Google Sheets
What are the details of your problem?
I am a teacher and I want to use Python to create a worksheet for my students. I have a vocabulary PDF with content like this:
do your best duː jɔː best
33, 81
do ...
1
vote
3
answers
110
views
CSV iteration addition algorithm Python
I am currently trying to write an algorithm that reads in a CSV file, gets a list of names from column 0, a list of hours worked from column 6, then iterate through a list of staff names, if a name is ...
1
vote
1
answer
210
views
How to export a tbl_duckdb_connection object to CSV from duckdb without collect()?
I have a dataset (originally large CSV) that I filtered using duckdb and dbplyr.
This is a small script that get to my idea :
library(duckdb)
library(DBI)
library(dplyr)
library(dbplyr)
...
2
votes
1
answer
115
views
How to bulk load CSV file in mySQL workbench without import wizard?
I am trying to write a stored procedure to import CSV data from local system to mySQL tables. I do not want to use import wizard as I want SP to create, import and display time taken to import.
...
2
votes
1
answer
220
views
`duckdb_read_csv` not working when there are double quotes ""example"" in CSV
I have a large CSV that was generated from GBIF (So modifying the raw csv is not what I'm looking for). Within the CSV, there are lines where there are double double quotes (e.g., "Henry "&...
0
votes
1
answer
69
views
Python Convert List Image to multiple PDF [closed]
In the case
I have a folder of image list and list name file with csv
I want to write a python script to convert list of PNG's to multiple pdf file with name file of pdf from csv file at column D
like ...
6
votes
2
answers
390
views
Pandas does not fail, warn, or skip when rows have more columns than the header
I'm new to Python and to Pandas, and I am desperately trying to understand how or why this is happening.
I have a CSV file with some data, which has some rows which have extra commas , which are not ...
0
votes
1
answer
121
views
How to make Microsoft Excel load Unicode characters correctly when open double clicking CSV file
Note: This issue is specific to Microsoft Excel. It works fine on LibreOffice without any problems.
I am developing a React application where I need to let the user download some table data upon a ...
1
vote
1
answer
119
views
C# ExcelDataReader reading foreign characters incorrectly from CSV
I created a console application in C#. The program refreshes host names in my database while reading data from an Excel sheet using ExcelDataReader.
The Excel file is saved in .CSV format. One of the ...
0
votes
3
answers
158
views
How do I populate a column in a PowerShell array from a Groups column in a CSV by searching for a matching Name of a corresponding user?
I have a PowerShell script that works up to a certain point that imports two CSV files, CSV #1 and CSV #2, and creates a new array ($content3) with input from both CSV files. I need to populate the ...
0
votes
2
answers
171
views
How do I access specific keys and values from CSV files?
I have a CSV file keeping track of people's score in a game. I read it with csv.DictReader().
How do I create a dictionary with the keys being the names and the value being the scores?
CSV file ...
1
vote
0
answers
56
views
fast-csv append writing headers to last row
I'm appending data to a CSV file from a JSON rest API call. However, I'm finding whenever I write too it, it's adding the headers to the last item in the list.
My code is as follows
const csv = ...
1
vote
2
answers
242
views
Pandas DtypeWarning "Columns have mixed types" for large CSV file (no error with one less line)
I am using Pandas (v2.2.3) to read/load a (relatively large) CSV file using read_csv(). The full file has about 500k lines.
The function throws a DtypeWarning stating that "Columns have mixed ...
1
vote
1
answer
37
views
Neo4J (5.14.0): add labels based on values in CSV file
I'm trying to create nodes with one set label (:object) and two additional labels that are based on values in the following CSV file:
Object_ID
Classification
Hierarchy_name
Object_type
O0010
...
2
votes
2
answers
147
views
Am I correctly generating a list of randomly assigned pairs with exclusions in python?
I have an array of names and roles of people within a company:
Example array:
names_and_titles = [
("Samantha Reyes", "Innovation", "Product Owner"),
("Ethan ...
3
votes
1
answer
83
views
How to use emptyValue option in pyspark while reading a csv file?
According to docs of csv options:
Property Name
Default
Meaning
emptyValue
(for reading), "" (for writing)
Sets the string representation of an empty value.
But it doesn't seem to work:
with ...
2
votes
1
answer
150
views
How to read empty string as well as NULL values from a csv file in pyspark? [duplicate]
Read spark csv with empty values without converting to null doesn't answer this one because:
That one's scala and this is pyspark.
Scala solution .option("nullValue", null) translates to ...
4
votes
3
answers
136
views
Why is pandas not formatting dates with date_format?
Why is pandas not formatting dates with date_format argument of to_csv?
pandas.DataFrame([datetime.datetime.now().date()]).to_csv(date_format="%Y %b")
',0\n0,2025-07-31\n'
4
votes
5
answers
299
views
How to merge two CSV files based on matching values in different columns and keep unmatched rows with placeholders?
I'm working on a data cleaning task and could use some help. I have two CSV files with thousands of rows each:
File A contains product shipment records.
File B contains product descriptions and ...
0
votes
1
answer
102
views
Duplicate Headers and Rows in CSV File When Exporting PowerShell Log
I'm using the following PowerShell script to remove self-signed certificates from a server. Based on the outcome, I log the results to a CSV file. However, I'm encountering two issues:
Duplicate ...
-3
votes
1
answer
98
views
Export to CSV issue only return the last data [closed]
I have my script here:
import json
import csv
from pathlib import Path
import os
# Path
downloads = os.path.expanduser("~/Downloads")
json_path = os.path.join(downloads, "...
1
vote
3
answers
166
views
How can I read multiple csv files from within multiple zip files with sometimes different columns using R fread?
I'm trying to read in a lot of data as efficiently as possible; the data are in about 1,400 CSV files within 6 individual ZIP files. The CSV files are all similar timeseries data with the same columns,...
2
votes
2
answers
146
views
Issue with Number Formatting When Converting CSV to Excel Using Pandas
I'm having trouble with a Python script that converts CSV files to Excel format using pandas. The issue is that numbers in scientific notation in the CSV file are not being correctly interpreted as ...