| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-09 | |||
| 16:35:44 | melwitt | spatel: it's not uncommon but it's not desirable. I have seen it many times, related to environmental issues like network disruptions. it's also similar to the DBConnectionError you pasted earlier, that one was about the pymysql client failing to be able to connect to the database, these rabbit errors are showing connection reset or closed trying to connect to rabbitmq | |
| 16:37:05 | spatel | These controller and compute node on same rack | |
| 16:37:19 | spatel | let me restart my whole rabbitMQ cluster | |
| 16:38:32 | spatel | when i delete vm they getting stuck in deleting state | |
| 17:14:36 | dansmith | melwitt: ack, thanks | |
| 17:39:55 | opendevreview | Artom Lifshitz proposed openstack/nova master: docs: Explicitly state lack of support for node renaming https://review.opendev.org/c/openstack/nova/+/795607 | |
| 17:58:43 | artom | Huh, looks like the docs gate is broken on nova/crypto.py | |
| 17:58:48 | artom | ^^ somehow failed it | |
| 18:03:10 | artom | Actually, just pep8 in general is broken: https://zuul.opendev.org/t/openstack/build/0b3b9315c9cb4d489f2ffb1355145a99 | |
| 18:04:32 | artom | Ah, it's mypy... | |
| 18:16:14 | lyarwood | artom: the fix for that is in the gate | |
| 18:16:44 | artom | lyarwood, oh. And here I was filing a bug and writing a fix. | |
| 18:16:44 | lyarwood | https://review.opendev.org/c/openstack/nova/+/795533 | |
| 18:17:41 | lyarwood | Always enjoyable when that happens ^_^ | |
| 18:18:23 | artom | I mean, no mailing list message, no bug filed, nothing :P | |
| 18:18:57 | artom | Hrmm, I wonder if my fix could stiff have value | |
| 18:21:26 | artom | I guess not, since we'll need to manually update requirements anyways? | |
| 18:21:54 | artom | I did it by just adding `python -m mypy --install-types` to tools/mypywrap.sh | |
| 18:23:31 | lyarwood | Yeah I think stephenfin tried that earlier | |
| 18:23:40 | lyarwood | but it wasn't enough in the gate for some reason | |
| 18:24:12 | artom | I guess because lower-constraints.txt? | |
| 18:42:59 | cz3 | This is probably a trivial question... I wanted to import Nova modules into interactive Python shell, but I cannot figure out how to load the config from default nova.conf. Is `CONF = nova.conf.CONF` supposed to handle this? | |
| 18:43:54 | cz3 | I wanted to play around with objects module through interactive shell but the lack of loaded config is blocking me | |
| 18:45:45 | artom | cz3, never tried it myself, but looking at the code, yeah, import nova.conf; CONF = nova.conf.CONF should do it | |
| 18:49:26 | cz3 | yeah, that's what I tried... doesn't seem to work. looks like it is still trying to use the defaults. I assume that because it tries to look up things in sqlite instead of mysql that I have in nova.conf | |
| 18:51:00 | artom | Maybe it's not finding the config file, and using the defaults... | |
| 18:52:30 | artom | In fact, if you're importing it via Python (and not via running a Python executable on the command line), I'm not even sure how it finds the config file to use... | |
| 18:54:21 | artom | I *think* you want to do CONF = nova.conf.CONF(project='nova') in such a case... | |
| 18:54:29 | artom | And then it'll look for nova.conf in the "normal" places | |
| 18:56:06 | cz3 | yes, I will poke around the nova.conf.CONF, maybe it needs some additional parameters when it is imported into interactive shell | |
| 18:56:29 | sean-k-mooney | cz3: you need to initalise the conf form the file too | |
| 18:56:40 | sean-k-mooney | if you just import it it will get the default form code | |
| 18:57:42 | sean-k-mooney | we do this https://github.com/openstack/nova/blob/master/nova/config.py#L94-L98 | |
| 18:57:53 | sean-k-mooney | where conf is CONF = nova.conf.CONF | |
| 19:02:02 | sean-k-mooney | all the command lets initalise it here https://github.com/openstack/nova/blob/d64edd3da2336a5c7c8f69cced45272cbaf638a9/nova/cmd/status.py#L374 | |
| 19:02:08 | sean-k-mooney | well liek ^ | |
| 19:03:24 | sean-k-mooney | so "import nova.conf; from nova import config; CONF = nova.conf.CONF; config.parse_args(sys.argv)" | |
| 19:03:41 | sean-k-mooney | cz3: if you do that i think it should work | |
| 19:04:59 | sean-k-mooney | although im not sure you need to initalise rpc or configre the db | |
| 19:05:21 | cz3 | in this case I just need the db access | |
| 19:06:49 | sean-k-mooney | which looks like its jsut sqlalchemy_api.configure(CONF) | |
| 19:07:26 | cz3 | sean-k-mooney: config.parse_args did the trick | |
| 19:07:31 | cz3 | thank you! | |
| 19:07:31 | sean-k-mooney | in any case i would use https://github.com/openstack/nova/blob/master/nova/cmd/manage.py as a reference | |
| 19:07:43 | sean-k-mooney | no worries | |
| 19:08:03 | sean-k-mooney | CONF(argv[1:], | |
| 19:08:05 | sean-k-mooney | project='nova', | |
| 19:08:07 | sean-k-mooney | version=version.version_string(), | |
| 19:08:09 | sean-k-mooney | default_config_files=default_config_files) | |
| 19:08:11 | sean-k-mooney | that is likely the bit you really needed | |
| 19:08:37 | sean-k-mooney | that is invokeing the gloabl constuctor for the singelton | |
| 19:09:11 | cz3 | I actually aim to add something to nova-manage and send it over for review, just wanted to test few things beforehand | |
| 19:09:35 | cz3 | I emailed the openstack-discuss list last week about this, it was re: request_specs regeneration | |
| 19:09:56 | cz3 | where I wanted to find a way to sync extra_specs from flavor data into existing instances | |
| 19:10:33 | sean-k-mooney | oh yes i remember | |
| 19:10:39 | cz3 | long time ago I did something like this by directly appending stuff into request_spec json in database, but this time I want to do it the proper way | |
| 19:13:20 | cz3 | I know that flavors are supposed to be immutable etc. but in this case I just want the existing request_specs to reflect the actual reality and not make the scheduler migrate instances where it was not supposed to after underlying flavor was updated with extra_specs | |
| 19:15:05 | cz3 | I will try to add that to nova-manage and send it over to review via gerrit, maybe it will be good enough to get upstreamed | |
| 19:15:54 | sean-k-mooney | the main consurn we woudl have it that such an update can lead to the vm being broken or on an invalid host also | |
| 19:16:30 | sean-k-mooney | on one hand nova manage is admin only and require direct acess to the server hosting the cloud to get the configs | |
| 19:16:59 | sean-k-mooney | but from a downstream product point of view im not sure we would support customer executing this command | |
| 19:17:16 | sean-k-mooney | we may be with the caveat tha that if you break things you fix them | |
| 19:17:33 | sean-k-mooney | its something we would have to think about | |
| 19:18:01 | cz3 | yeah, I do share the concern, I don't want to send something that would end up breaking deployments for other people if used improperly | |
| 19:18:35 | sean-k-mooney | you proably would need to run som of the flavor/image compatiabity checks before doing the update to make it safe | |
| 19:18:53 | sean-k-mooney | our add a --yes-i-really-really-mean-it flag | |
| 19:19:14 | sean-k-mooney | anyway o/ time to go have dinner | |
| 19:19:42 | cz3 | sure | |
| 19:20:05 | cz3 | thanks again! | |
| 21:57:38 | opendevreview | melanie witt proposed openstack/nova stable/ussuri: zuul: Start to migrate nova-live-migration to zuulv3 https://review.opendev.org/c/openstack/nova/+/795432 | |
| 21:57:39 | opendevreview | melanie witt proposed openstack/nova stable/ussuri: Remove broken legacy zuul jobs https://review.opendev.org/c/openstack/nova/+/795374 | |
| 22:07:55 | tosky | melwitt: an supersquash! Would it make sense to change the first line of the commit message? It is more than just that single commit it was taken from | |
| 22:21:29 | opendevreview | melanie witt proposed openstack/nova master: api: Log correct client IP if load balancer in use https://review.opendev.org/c/openstack/nova/+/786766 | |
| 22:25:02 | gmann | stephenfin: can you check if anything else you want to add in this spec, leaving +w to you https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| #openstack-nova - 2021-06-10 | |||
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Add functional regression test for bug 1853009 https://review.opendev.org/c/openstack/nova/+/695012 | |
| 00:02:37 | opendevmeet | bug 1853009 in OpenStack Compute (nova) ussuri "Ironic node rebalance race can lead to missing compute nodes in DB" [High,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard) | |
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/c/openstack/nova/+/695187 | |
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/c/openstack/nova/+/695188 | |
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/694802 | |
| 00:02:38 | opendevreview | melanie witt proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/695189 | |
| 06:39:39 | masterpe[m] | I have more resources declined in the placement.allocations table then actual in use. I see some instance ID there that are deleted. | |
| 06:39:56 | masterpe[m] | What shall I do with them? | |
| 06:46:16 | frickler | masterpe[m]: seems this tool was made for you https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement-audit | |
| 06:46:53 | frickler | or maybe the heal-allocations above | |
| 06:52:57 | masterpe[m] | nice thanks | |
| 08:06:14 | lyarwood | kashyap: https://review.opendev.org/c/openstack/nova/+/795533/5#message-3e7b1dc4d3f8d22b2c9a55637c7366199d00eb56 - I'll write this up in a Nova bug and likely libvirt bug later today but would you mind scanning this if you get a chance? | |
| 08:06:37 | kashyap | lyarwood: Mornin; /me clicks | |
| 08:07:30 | kashyap | lyarwood: Ah, you're debugging the informative error, "reason=failed" | |
| 08:07:45 | lyarwood | there's two parts really | |
| 08:07:51 | lyarwood | yeah the reason=failed thing | |
| 08:08:18 | lyarwood | and the way that the call to virDomainMigrateToURI3 from the source doesn't pick up the failure on the dest | |
| 08:09:23 | kashyap | Hmm, yikes | |
| 08:09:25 | lyarwood | I'm assuming that the dest tears down the connection between the two leading to the eventual `unable to connect to server` error | |
| 08:10:09 | lyarwood | instead of the source being told the migration has failed etc | |
| 08:10:47 | kashyap | lyarwood: This is the dest, right: https://zuul.opendev.org/t/openstack/build/f3b829801901417c9310ad5cc5a0e886/log/controller/logs/libvirt/libvirtd_log.txt | |
| 08:10:54 | lyarwood | yeah | |
| 08:11:15 | kashyap | Is it just me or is it loading deadly slow? I just want to pull down the entire raw file | |
| 08:11:50 | lyarwood | yeah these files are pretty large, I typically pull them down now | |
| 08:12:02 | kashyap | My browser is hung here (probably it goes into MBs). Do I clicck on the "View log" to get the raw link? | |
| 08:12:15 | lyarwood | it's awkward as the raw links don't use .gz at the end of the file names | |