You need to edit your question to show (1) the code that you used (2) the full error and traceback (3) the url that is involved (4) what is the unicode character that you are trying to encode as gbk You seem to have somehow obtained unicode characters from the raw bytes in the the html content -- how? what encoding is specified in the html content? Then (I guess) you are trying to write the unicode characters to a file, endcoding the unicode as gbk. During this process, you got an error something like this:
If the raw bytes in the html content were not encoded in gbk, then it is quite possible that you have some unicode characters that can't be represented in gbk. In that case you may like to encode your results using the original encoding, or encode them in gb18030 which can take any unicode character. Another possibility is that you have mangled the raw bytes or the unicode somehow. I certainly hope that your regex machinations have been done on the unicode and not on some variable-length-character encoding like gb2312, gbk, etc. Update: Here is your code snippet:
From that I've had to deduce: As my sys.defaultencoding is NOT 'gbk', I replaced your last 3 lines with Observations: (1) website has charset=utf-8, decodes OK with utf-8
So the web page contains Unicode characters which are not mapped in gbk. Options are (1) encode with 'gbk' but use the 'replace' option |
|