I’m running the following code using numpy arrays, I get a MemoryError in Ubuntu, while the same code runs on Mac OSX. (Pagination is automatically setup in Mac) This process consumes around 30 GB. Original post here.
I’m trying to replace
numpy array with another structure which doesn’t consume so much memory and improve the
def create_sequences(tokenizer, max_length, descriptions, photos): """Creates sequences of images, input sequences and output words for an image. X1, X2 (text sequence), y (word) photo startseq, little photo startseq, little, girl photo startseq, little, girl, running photo startseq, little, girl, running, in photo startseq, little, girl, running, in, field photo startseq, little, girl, running, in, field, endseq :param tokenizer: :param max_length: :param descriptions: :param photos: :return: """ X1, X2, y = , ,  # Walk through each image identifier. for desc_key, desc_list in descriptions.iteritems(): # Walk through each description for the image. for desc in desc_list: # Encode the sequence. seq = tokenizer.texts_to_sequences([desc]) # Split one sequence into multiple X,Y pairs. for i in range(1, len(seq)): # Split into input and output pair. in_seq, out_seq = seq[:i], seq[i] # Pad input sequence. in_seq = pad_sequences([in_seq], maxlen=max_length) # Encode output sequence out_seq = to_categorical([out_seq], num_classes=vocab_size) # Store. X1.append(photos[desc_key]) X2.append(in_seq) y.append(out_seq) print len(X1), len(X2), len(y) print type(X1) #return array(X1), array(X2), array(y)
Dataset: 6000 train images. Descriptions: train=6000 Vocabulary Size: 7579 Photos: train=6000 Description Length: 34 Preparing text sequences for training. 306404 306404 306404 <type 'numpy.ndarray'>
✓ Extra quality
ExtraProxies brings the best proxy quality for you with our private and reliable proxies
✓ Extra anonymity
Top level of anonymity and 100% safe proxies – this is what you get with every proxy package
✓ Extra speed
1,ooo mb/s proxy servers speed – we are way better than others – just enjoy our proxies!
USA proxy location
We offer premium quality USA private proxies – the most essential proxies you can ever want from USA
Our proxies have TOP level of anonymity + Elite quality, so you are always safe and secure with your proxies
Use your proxies as much as you want – we have no limits for data transfer and bandwidth, unlimited usage!
Superb fast proxy servers with 1,000 mb/s speed – sit back and enjoy your lightning fast private proxies!
99,9% servers uptime
Alive and working proxies all the time – we are taking care of our servers so you can use them without any problems
No usage restrictions
You have freedom to use your proxies with every software, browser or website you want without restrictions
Perfect for SEO
We are 100% friendly with all SEO tasks as well as internet marketing – feel the power with our proxies
Buy more proxies and get better price – we offer various proxy packages with great deals and discounts
We are working 24/7 to bring the best proxy experience for you – we are glad to help and assist you!