Best practices for uploading large book collection via Calibre?

[Gadgetguy]

Hello all,


just got this morning my brand new Forma.

After some challenges (the Kobo website is down so I had to hack the registration) the Kobo is up and running.

I started to transfer my book collection using Calibre.

And here I have a challenge: when I transfer a large number of books - I alway get an error in Calibre "Error communicating with device" and the transfer fails.

I have already removed the Kobo utilities plug-in and I have plugged the Kobo directly into my laptop (so no USB hub) - but I still get this error.

Yes, I can transfer 10 books at a time - but with many hundreds of books that is a pain.

Is there a recommended way for transferring hundreds of books in one go?

Thank you for any suggestions!

(The communication itself works well between laptop and Kobo - I copied 2 GB of PDF to the unit with copy - paste with no problem)

[JSWolf]

Originally Posted by Gadgetguy

Hello all,


just got this morning my brand new Forma.

After some challenges (the Kobo website is down so I had to hack the registration) the Kobo is up and running.

I started to transfer my book collection using Calibre.

And here I have a challenge: when I transfer a large number of books - I alway get an error in Calibre "Error communicating with device" and the transfer fails.

I have already removed the Kobo utilities plug-in and I have plugged the Kobo directly into my laptop (so no USB hub) - but I still get this error.

Yes, I can transfer 10 books at a time - but with many hundreds of books that is a pain.

Is there a recommended way for transferring hundreds of books in one go?

Thank you for any suggestions!

(The communication itself works well between laptop and Kobo - I copied 2 GB of PDF to the unit with copy - paste with no problem)

I suggest you log out of your Kobo and register properly via the website or you cannot sync and you cannot use Overdrive. I don't know if you'll be able to use Dropbox or not without a proper registration.

Anyway, are you transferring ePub or KePub? Have you tried both to see which one you want to use for side loaded eBooks? Once you figure this out, if you do go for KePub, you'll want to use the KoboTouchExtended driver convert ePub to KePub and load them on your Forma.

Try side loading a smaller amounts and see if that works like maybe 10 at a time.

[Gadgetguy]

Originally Posted by JSWolf

I suggest you log out of your Kobo and register properly via the website or you cannot sync and you cannot use Overdrive. I don't know if you'll be able to use Dropbox or not without a proper registration.

Anyway, are you transferring ePub or KePub? Have you tried both to see which one you want to use for side loaded eBooks? Once you figure this out, if you do go for KePub, you'll want to use the KoboTouchExtended driver convert ePub to KePub and load them on your Forma.

Try side loading a smaller amounts and see if that works like maybe 10 at a time.

Thank you, it works now. One of the calibre plugins was the problem. (Not sure which one - I had several and I removed all).
I am unable to properly set up as Kobo is currently down...
Maybe will redo when Kobo is again operational.

[davidfor]

Originally Posted by Gadgetguy

Thank you, it works now. One of the calibre plugins was the problem. (Not sure which one - I had several and I removed all).

I'm curious about what gives you that impression. None of them should have caused the error you reported. The only one that is actually involved in transferring the books is the KoboTouchExtended driver. But, it is a bit hard to know what happened there, as you didn't post the detailed error message. That shows more about what went wrong and when. Also, what happened with the connection to the PC? Did it disconnect as well?

From the point of view of loading a new device, I have sideload several thousand from calibre in one go. I usually limit it to batches of 1000, but that's as much about patience and wanting to use the device as anything else. I haven't had a problem doing this in the last few years.

I am unable to properly set up as Kobo is currently down...
Maybe will redo when Kobo is again operational.

You were pretty unlucky to need to do this just when Kobo was doing some maintenance. But, if you are planning to purchase books from Kobo, or use a library, you will need to sign in properly. But, what you did is unlikely to have caused the problems with calibre. Unless you also managed to corrupt the database at the same time.

[Gadgetguy]

Originally Posted by davidfor

I'm curious about what gives you that impression. None of them should have caused the error you reported. The only one that is actually involved in transferring the books is the KoboTouchExtended driver. But, it is a bit hard to know what happened there, as you didn't post the detailed error message. That shows more about what went wrong and when. Also, what happened with the connection to the PC? Did it disconnect as well?

Hi, I have transferred a big quantity of PDF's - (over 2 GB) and they went through in one try, no issues (hence the PC to Kobo hardware connection seems solid)

For the Calibre plugins: with the plugins installed I was not able to transfer more than 10 or so books in one run - otherwise I got errors.

With the plugins removed I was able to transfer all ebooks in one run, no problem (500+ books).

That lead me to beleive that one of the plugins was the culprit.

I did it now again (as I did a factory reset and proper setup after properly activating with Kobo this morning) and Calibre transferred again all books at once with no issue (but I have not reinstalled any plugin yet).

Will start to reinstall the plugins one by one when I have some time to determine which one is the culprit.

[PeterT]

I wonder if there might be a timeout issue when using the Kobo Touch Extended driver to transfer a large number of books to the device, and it having to massage the ePubs to create kEpubs during the transfer?

[Cdesja5]

I just had the same issue with my Kobo Glo HD (with new firmware) and am now trying with 1000 books at a time. Am uploading epubs.

[davidfor]

Originally Posted by PeterT

I wonder if there might be a timeout issue when using the Kobo Touch Extended driver to transfer a large number of books to the device, and it having to massage the ePubs to create kEpubs during the transfer?

There shouldn't be. It does the work to transform the books on local disk. Then it copies the book to the device and possibly does a database update. None of that is time sensitive. Without know where and what the error was, it's pretty hard to guess.

[Cdesja5]

No go when I tried to add 1000 books at a time to my Glo HD - I therefore removed the Kobo add-ons and then added them back again. It seems the problem is (according to the job list) getting the list of books on device - that always ends in an error.
IOError: disk I/O error

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 90, in run
File "site-packages\calibre\gui2\device.py", line 513, in _books
File "site-packages\calibre\devices\kobo\driver.py", line 1964, in books
File "c:\t\t\apsw-w7591s5i\src\cursor.c", line 236, in resetcursor
IOError: IOError: disk I/O error

I have since (without re-starting Calibre) just tried 399 books and got an error. So for the book transfer, it seems the problem is a corrupted file (which I have removed). I can't figure out what the issue getting the kobo's book list into Calibre is though.
ADDITIONAL INFO: I removed the file that was corrupted according to the error message and got the same message with the next file in the list when I tried again... My Glo is now only good for 300 or so books - so disappointed - it's been used very sporadically and with kid gloves and now seems "broken" because of this update.

[Error 1392] The file or directory is corrupted and unreadable: u'E:\\Holiday, Jenny'

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 90, in run
File "site-packages\calibre\gui2\device.py", line 594, in _upload_books
File "calibre_plugins.kobotouch_extended.device.driver" , line 403, in upload_books
File "site-packages\calibre\devices\kobo\driver.py", line 2132, in upload_books
File "site-packages\calibre\devices\usbms\driver.py", line 324, in upload_books
File "site-packages\calibre\devices\usbms\device.py", line 998, in create_upload_path
File "os.py", line 157, in makedirs
WindowsError: [Error 1392] The file or directory is corrupted and unreadable: u'E:\\Holiday, Jenny'

[Grins]

Are you sure that you don't exceed the capacity of the internal memory? I've got a similar problem some time ago ...

[Cdesja5]

I reset it. It's empty. I have tried several times to add books from Calibre (with a factory reset and update each time) and still get the error messages. I had never had issues previously loading books to the Glo, nor do I have problems with my Mini (for which no firmware update will be supplied).

[JSWolf]

Originally Posted by Cdesja5

No go when I tried to add 1000 books at a time to my Glo HD - I therefore removed the Kobo add-ons and then added them back again. It seems the problem is (according to the job list) getting the list of books on device - that always ends in an error.
IOError: disk I/O error

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 90, in run
File "site-packages\calibre\gui2\device.py", line 513, in _books
File "site-packages\calibre\devices\kobo\driver.py", line 1964, in books
File "c:\t\t\apsw-w7591s5i\src\cursor.c", line 236, in resetcursor
IOError: IOError: disk I/O error

I have since (without re-starting Calibre) just tried 399 books and got an error. So for the book transfer, it seems the problem is a corrupted file (which I have removed). I can't figure out what the issue getting the kobo's book list into Calibre is though.

[Error 1392] The file or directory is corrupted and unreadable: u'E:\\Holiday, Jenny'

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 90, in run
File "site-packages\calibre\gui2\device.py", line 594, in _upload_books
File "calibre_plugins.kobotouch_extended.device.driver" , line 403, in upload_books
File "site-packages\calibre\devices\kobo\driver.py", line 2132, in upload_books
File "site-packages\calibre\devices\usbms\driver.py", line 324, in upload_books
File "site-packages\calibre\devices\usbms\device.py", line 998, in create_upload_path
File "os.py", line 157, in makedirs
WindowsError: [Error 1392] The file or directory is corrupted and unreadable: u'E:\\Holiday, Jenny'

It looks like the SD card in your Glo HD is corrupt. That would explain the I/O errors. You can request an SD image from davodfor (PM) and using a new SD card, restore the image, open the Glo HD, swap the SD card and be on your way. There are directions how to restore the image in the Kobo Developers forum.

Here is the link to the forum thread you'll need to read.

https://www.mobileread.com/forums/sh...d.php?t=226719

[Gadgetguy]

Originally Posted by JSWolf

It looks like the SD card in your Glo HD is corrupt. That would explain the I/O errors. You can request an SD image from davodfor (PM) and using a new SD card, restore the image, open the Glo HD, swap the SD card and be on your way. There are directions how to restore the image in the Kobo Developers forum.

Here is the link to the forum thread you'll need to read.

https://www.mobileread.com/forums/sh...d.php?t=226719

Just for context - I had these errors with a brand new (well, refurb) Forma reader, and removing the plug in's solved it... So as far as I am concerned there is 100% an issue with the plug ins

[davidfor]

Originally Posted by Gadgetguy

Just for context - I had these errors with a brand new (well, refurb) Forma reader, and removing the plug in's solved it... So as far as I am concerned there is 100% an issue with the plug ins

Sorry, that is bullshit. You have stated this before with no real evidence. I don't even think you bothered to mention what plugins you thought were causing the problem. Even the use of the plural for the plugins is a problem as there is only one that could even remotely be causing a problem. And even that is doubtful as 95% of the code used is actually in the built-in KoboTouch driver. Nearly all of the extra code acts on file on your computer before they are sent to the device.

[davidfor]

Originally Posted by Cdesja5

No go when I tried to add 1000 books at a time to my Glo HD - I therefore removed the Kobo add-ons and then added them back again. It seems the problem is (according to the job list) getting the list of books on device - that always ends in an error.
IOError: disk I/O error

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 90, in run
File "site-packages\calibre\gui2\device.py", line 513, in _books
File "site-packages\calibre\devices\kobo\driver.py", line 1964, in books
File "c:\t\t\apsw-w7591s5i\src\cursor.c", line 236, in resetcursor
IOError: IOError: disk I/O error

That error is usually a sign that the database on the device is corrupt. It can sometime be fixed by restarting the device. But, usually you need to replace the database on the device either with a backup, or by signing out of the Kobo account on the device and back in again. The latter creates a new database and imports the sideloaded books again.

I have since (without re-starting Calibre) just tried 399 books and got an error. So for the book transfer, it seems the problem is a corrupted file (which I have removed). I can't figure out what the issue getting the kobo's book list into Calibre is though.

What was the error that said the book was corrupt? Unless you are using the KoboTouchExtended driver, calibre isn't doing anything that should trigger an error like this. Or are you saying the error happened on the device when it tried to import the book/

ADDITIONAL INFO: I removed the file that was corrupted according to the error message and got the same message with the next file in the list when I tried again... My Glo is now only good for 300 or so books - so disappointed - it's been used very sporadically and with kid gloves and now seems "broken" because of this update.

[Error 1392] The file or directory is corrupted and unreadable: u'E:\\Holiday, Jenny'

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 90, in run
File "site-packages\calibre\gui2\device.py", line 594, in _upload_books
File "calibre_plugins.kobotouch_extended.device.driver" , line 403, in upload_books
File "site-packages\calibre\devices\kobo\driver.py", line 2132, in upload_books
File "site-packages\calibre\devices\usbms\driver.py", line 324, in upload_books
File "site-packages\calibre\devices\usbms\device.py", line 998, in create_upload_path
File "os.py", line 157, in makedirs
WindowsError: [Error 1392] The file or directory is corrupted and unreadable: u'E:\\Holiday, Jenny'

That to me says the internal SD card is failing. It might be that the filesystem for the book partition on the device is badly corrupted. Either of these could also be the cause of the first error.

The firmware update is probably a coincidence. One of the things that happen is that the device is rebooted. If you don't do that very often, any issues with the database will come up at this point.

You could try a factory reset. This will format the book partition and take you through the setup. If it is just a problem with file system, it will be fixed. If it is a problem with the card, it might temporarily fix it, but will probably happen again.