Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-15
10:28:56 mdbooth But it previously used GLANCE_LIMIT_IMAGE_SIZE_TOTAL=10000 instead
11:02:02 mdbooth frickler: FWIW I've been in hacker mode on that config for a while (just look at the history!). I just disabled tempest and horizon which had accidentally become enabled again, and it seems to have completed. It's still using GLANCE_ENABLE_QUOTAS=False.
11:03:39 mdbooth Which is to say, if there's a dependency issue I'll bet it relates to tempest or horizon, but I haven't proven that.
11:29:38 frickler mdbooth: o.k., at least I could reproduce your failure with GLANCE_LIMIT_IMAGE_SIZE_TOTAL being set
11:50:49 frickler mdbooth: nice one, this actually only fails consistently with DEVSTACK_PARALLEL=False
11:52:25 frickler with async, https://github.com/openstack/devstack/blob/82facd6edf7cefac1ab68de4fe9054d7c4cb50db/stack.sh#L1107 runs in the background and write_clouds_yaml in L1122 has a fair chance of being fast enough
11:52:33 frickler dansmith: ^^
11:56:05 mdbooth frickler: Oh, wow! I only turned that on temporarily to rule it out as the potential cause of another issue!
12:27:01 kashyap mdbooth: TIL, "tpl" extension
12:27:27 mdbooth kashyap: Not mine in this case, but I'm pretty sure I've used it before.
12:27:44 kashyap (From your link. Probably it's just a convenient reference to refer to that YAML file as a "template")
12:27:56 kashyap mdbooth: I see
12:56:27 gibi sean-k-mooney: about stoping the services. you are right we are doing it already. that does not stop all the eventlets the service spawnd. I also tried to iterate all the eventlets and and call throw() on them to stop them but that did not help either.
12:57:10 gibi kashyap: thanks for the links, I added it as weekend reading :)
12:57:46 kashyap gibi: No prob. (It took 8 gruelling revisions. :D. But I always become a bit of a better person after writing for LWN)
12:57:56 sean-k-mooney gibi: ya, i have your review open on my other monitor. the more i read over it and look at it the more compleing it becomes.
12:58:21 sean-k-mooney gibi: its a little non obviious at first glance why we have to do this but its a nice solution when yuou did into it
12:58:29 gibi kashyap: I follow LWN but not a subscriber. I think it is a prestige to write there :)
12:59:13 kashyap gibi: I realize not everyone has a subscription; Red Hat has a group sub. Hence I created a "subscriber link", as I posted it in a community channel.
12:59:16 gibi sean-k-mooney: would be better to kill eventlets at the end of each testcase, but I did not find a way to do it
12:59:27 gibi kashyap: yeah I see and I thank you for it
12:59:42 kashyap No prob at all. (And sorry for the plug.)
13:00:23 kashyap But the main idea of essential vs. accidental complexity comes from the famous 1986 paper called "No Silver Bullet" by Fred Brooks - https://en.wikipedia.org/wiki/No_Silver_Bullet
13:00:53 sean-k-mooney gibi: well there might be a way to do it if we modifed the test setup so that each test used a seperate greenpool then we could stop all eventlets in the pool and discard it at the end of the test
13:01:07 kashyap (So it was nice to see concrete examples of it in QEMU.)
13:01:48 sean-k-mooney to do that i think we would have to modify the nova service deffintion and possible nova utils to use a non default eventlet pool
13:02:13 sean-k-mooney but if we did that we could extend the kill function to terminate the pool
13:03:38 frickler mdbooth: I wanted to move the write_clouds_yaml earlier anyway in https://review.opendev.org/c/openstack/devstack/+/780417, I guess I can just do that step in its own patch to fix your issue
13:05:02 gibi kashyap: ohh yeah essential and accidental complexity I like those topics
13:05:23 mdbooth frickler: I'd appreciate it
13:05:30 kashyap gibi: Yeah; the idea goes back 2000 years ago! (Aristotle++)
13:05:34 gibi ohh
13:05:37 gibi I did not know that
13:06:02 kashyap I linked to it in the intro too :)
13:06:22 mdbooth eventlet--
13:06:33 kashyap mdbooth: Heh, what a contrasting negative karma
13:06:41 kashyap (Sorry for your pain)
13:07:00 mdbooth I only have the scars now, and occasionally the nightmares.
13:07:03 gibi sean-k-mooney: if terminating the pool also just calls greenlet.throw() then that would have the same problem as I had when I manually called that at the end of the test on each greenlet
13:08:22 frickler mdbooth: https://review.opendev.org/c/openstack/devstack/+/814142
13:08:56 mdbooth It blows my mind that at some point there was a meeting and somebody said: "You know what, lets just monkey patch everything and replace it all with our own stuff, what could go wrong?". And somebody else in that meeting agreed with them, and they started doing it.
13:09:35 gibi mdbooth: I assume it was a single person project. :)
13:09:36 sean-k-mooney mdbooth: well the alternitive was to continue to use twisted so...
13:10:31 gibi our use threading until you scale too big where the overhead of threads are too much
13:10:55 gibi our use other than python without the GIL ;)
13:11:01 sean-k-mooney gibi: i was considering coudl we stop all the service we spawwned as greantreads and then etiehr call waitall to wait for the to finsih or loop over and call kill on the all running greenthreads
13:12:19 sean-k-mooney so stop service and call https://eventlet.net/doc/modules/greenpool.html#eventlet.greenpool.GreenPool.waitall or stop services and call https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.kill on all greentreads in the pool
13:12:31 gibi it is multiple eventlet per service, but yes, you are right. That should work. I did not call wait after throw, maybe that was the problem
13:12:55 gibi note that we not just have greenthreads, we have naked greenlets as well somehow
13:13:09 gibi I did not traced where they are coming from
13:15:23 sean-k-mooney i need to test something else today but i still think your current patch is likely a viable solution in the sort term and we could explore the green pool approche in parallel/after
13:25:02 bauzas folks, looking at the nova PTG agenda we have atm
13:25:41 bauzas it looks to me we don't have a lot of topics to discuss, so maybe we shouldn't have a schedule, ok ?
13:26:01 bauzas I'll just prioritize some topics
13:30:50 dansmith frickler: ah, need to wait for all those accounts to finish before write_clouds_yaml I guess huh?
13:30:51 sean-k-mooney we might want to keep one of the session free for an unconfrence/follow up dicussions
13:30:57 sean-k-mooney bauzas: ^
13:31:21 sean-k-mooney bauzas: but ya we could also just priortise the list and see how far we get each day
13:37:41 dansmith frickler: er, no I guess that's just writing static things out, so .. I'm not sure what the problem is (if any)
13:38:02 gibi bauzas: I suggest to frontload the important stuff and then just follow the etherpad. If we run out of topics then we are done :)
13:38:29 gibi sean-k-mooney: yeah I have to do other things too today so I have no chance to try the greenpool approach
13:48:31 bauzas gibi: yeah, for example, I'll move the melwitt's topic for unified limits above
13:54:14 gibi ack
14:05:50 mdbooth FYI: $ curl --compressed -H "X-Auth-Token: ${token}" -X GET https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13292/v2.1/images/$imageid/file | nbdcopy -- - [ qemu-nbd -f qcow2 capo-e2e-worker.qcow2 ]
14:06:31 mdbooth Ah, wrong channel. Maybe still interesting, though :)
14:08:50 mdbooth dansmith: The problem I was hitting was that we were trying to create the glance quotas before clouds.yaml had been created.
14:10:54 mdbooth And to be clear, I'm basically just cargo culting this local.conf. I have very little idea what's actually going on.
14:14:20 mdbooth Speaking of which, anybody ever seen: "The unit files have no installation config (WantedBy=, RequiredBy=, Also=, Alias= settings in the [Install] section, and DefaultInstance= for template units). This means they are not meant to be enabled using systemctl.". This failure seems to be non-deterministic, and unfortunately only happens in CI so I
14:14:20 mdbooth can't debug :(
14:52:50 dansmith mdbooth: ah, maybe I should move that in glance because I do it super early when the *glance* accounts are available, but I definitely need clouds.yaml
17:03:57 opendevreview Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519
21:08:11 opendevreview melanie witt proposed openstack/nova master: DNM Run against unmerged oslo.limit changes https://review.opendev.org/c/openstack/nova/+/812236
#openstack-nova - 2021-10-16
02:02:07 opendevreview melanie witt proposed openstack/nova master: DNM Run against unmerged oslo.limit changes https://review.opendev.org/c/openstack/nova/+/812236
#openstack-nova - 2021-10-18
07:24:32 bauzas good morning Nova
07:24:42 bauzas and happy PTG in the next hours
12:48:27 sean-k-mooney bauzas: are the nova topics starting today or tomorrow?
12:48:37 bauzas tomorrow
12:49:46 bauzas https://etherpad.opendev.org/p/nova-yoga-ptg
12:49:58 bauzas see also https://ptg.opendev.org/
12:50:03 sean-k-mooney bauzas: yep i have that open its reeally not clear based on that
12:50:29 sean-k-mooney although the live one is more clear sicne we donthave nova sessions today
12:50:41 sean-k-mooney as in https://ptg.opendev.org/ptg.html
12:51:07 bauzas sean-k-mooney: I just changed "Nova has meeting slots from Tuesday - Friday 13:00 - 17:00 UTC" to be a heading 3
12:51:55 sean-k-mooney ack i tought we werre gong to do 5 days which is why i asked with some corss project session today
12:52:06 sean-k-mooney but i see most of them are tommorow
12:52:56 bauzas yup, tbc I'll be on the TC/PTL session at 3pm UTC
12:53:23 sean-k-mooney i normally atend oslo or kolla stuff on mondays but i might look a neutron instead
16:21:45 opendevreview Merged openstack/nova master: db: Add migration to resolve shadow table discrepancies https://review.opendev.org/c/openstack/nova/+/805738
17:11:53 opendevreview Balazs Gibizer proposed openstack/nova master: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813419
17:16:46 opendevreview Merged openstack/nova master: Fix the wrong exception used to retry detach API calls https://review.opendev.org/c/openstack/nova/+/809934
19:13:38 NobodyCam Good Morning Nova Folks, crazy is there a way to recreate ironic node hypervisor records with out re-enrolling the node?
19:14:39 NobodyCam s/crazy/crazy question/
19:26:32 opendevreview Stephen Finucane proposed openstack/nova master: db: Enable auto-generation of API DB migrations https://review.opendev.org/c/openstack/nova/+/812144
19:26:33 opendevreview Stephen Finucane proposed openstack/nova master: db: Remove unused build_requests columns https://review.opendev.org/c/openstack/nova/+/812145
19:26:33 opendevreview Stephen Finucane proposed openstack/nova master: db: Remove legacy placement models https://review.opendev.org/c/openstack/nova/+/812146
19:26:34 opendevreview Stephen Finucane proposed openstack/nova master: objects: Stop querying the main DB for keypairs https://review.opendev.org/c/openstack/nova/+/812147
19:26:34 opendevreview Stephen Finucane proposed openstack/nova master: objects: Remove 'bandwidth' fields from notifications https://review.opendev.org/c/openstack/nova/+/812148
19:26:35 opendevreview Stephen Finucane proposed openstack/nova master: db: Remove models that were moved to the API database https://review.opendev.org/c/openstack/nova/+/812149
19:26:35 opendevreview Stephen Finucane proposed openstack/nova master: db: Remove models for removed services, features https://review.opendev.org/c/openstack/nova/+/812150
19:26:36 opendevreview Stephen Finucane proposed openstack/nova master: db: Remove nova-network models https://review.opendev.org/c/openstack/nova/+/812151

Earlier   Later