| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-09 | |||
| 15:15:54 | sean-k-mooney | e.g. if we had jsut defaulted to nova-spec or if it was intentional | |
| 15:16:31 | melwitt | sean-k-mooney: no, I'm not saying there was a decision, I had assumed something like that had occurred because of where the re-parent spec was proposed, but I was wrong | |
| 15:19:52 | gibi | melwitt: queued that up for review | |
| 15:20:24 | melwitt | thank you gibi | |
| 15:26:23 | opendevreview | Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575 | |
| 15:26:24 | melwitt | masterpe[m]: you need to look at the debug level logs in the placement api logs and the nova-scheduler logs to see details about why no valid host was found | |
| 15:28:39 | masterpe[m] | To solve it for now I enabled a node that was disabled/marked as spare. So I need to enable debug mode to troubleshoot? | |
| 15:32:42 | melwitt | masterpe[m]: to get full detail, yes. what you can do without debug mode is check whether there is a message like "no allocation candidates" in the nova-scheduler log, if you see that, it's likely cpu/ram/disk (as those resources are filtered by placement service) | |
| 15:33:42 | melwitt | if you don't see that, it means one or more of the nova side scheduler filters eliminated all hosts | |
| 15:34:11 | opendevreview | Balazs Gibizer proposed openstack/placement master: [DNM]: test consumer_type rollback due to consumer_gen conflict https://review.opendev.org/c/openstack/placement/+/795577 | |
| 15:36:31 | melwitt | gibi: oh heh, I had wondered if you saw my latest confusing comment. I was thinking the same thing this morning that it would be best to check what happens with a test. thanks for doing that | |
| 15:36:59 | gibi | melwitt: yeah, you are right the whole PUT /allocations/{uuid} call is not a single transaction | |
| 15:37:28 | gibi | so with the current implementation we can leak a consumer_type update from a rejected call :/ | |
| 15:39:31 | gibi | unfortunatly create a test case for a real race is too hard | |
| 15:39:43 | melwitt | gibi: ah, I see your new comment on the review. yeah ok, agree we need something higher up the stack then in order to not leak (or some other better idea yet to be thought of) :( I will work on it | |
| 15:40:32 | gibi | it is hard :/ | |
| 15:42:25 | melwitt | yeah, would require faking things with mocks at best. I didn't explore that yet so not sure how messy or possible | |
| 15:46:49 | spatel | sean-k-mooney periodically i am seeing this issue on my nova logs not sure if this is bug or not - http://paste.openstack.org/show/806496/ | |
| 15:51:45 | melwitt | spatel: that error means connection to the database failed while trying to write a service record update (heartbeat), not a bug. error being emitted from pymysql client | |
| 15:52:41 | spatel | melwitt today i said terraform to build 20 vm and 4 vm failed and stick in BUILD process, when i check their compute nodes i found that error | |
| 15:55:44 | melwitt | spatel: that paste wouldn't be directly related to a build failure (that paste is a service record update for a service heartbeat) so you'd need to look at which errors are related to the build fail, if any | |
| 16:03:13 | spatel | let me see, may be issue of rabbitmq in that case | |
| 16:05:41 | melwitt | dansmith: someone linked this review in the freenode channel and I thought you might have opinion about it. it's a proposal to allow config of rbd direct snapshot to enable an operator to disallow the fast clone, with the use case being the operator is migrating the backend and the fast clone wouldn't honor the glance backend config (I don't fully understand this part) https://review.opendev.org/c/openstack/nova/+/794837 | |
| 16:30:53 | spatel | melwitt also noticed periodically rabbitMQ connection reset error from nova-api - http://paste.openstack.org/show/806498/ | |
| 16:31:06 | spatel | is this a normal behavior ? | |
| 16:35:44 | melwitt | spatel: it's not uncommon but it's not desirable. I have seen it many times, related to environmental issues like network disruptions. it's also similar to the DBConnectionError you pasted earlier, that one was about the pymysql client failing to be able to connect to the database, these rabbit errors are showing connection reset or closed trying to connect to rabbitmq | |
| 16:37:05 | spatel | These controller and compute node on same rack | |
| 16:37:19 | spatel | let me restart my whole rabbitMQ cluster | |
| 16:38:32 | spatel | when i delete vm they getting stuck in deleting state | |
| 17:14:36 | dansmith | melwitt: ack, thanks | |
| 17:39:55 | opendevreview | Artom Lifshitz proposed openstack/nova master: docs: Explicitly state lack of support for node renaming https://review.opendev.org/c/openstack/nova/+/795607 | |
| 17:58:43 | artom | Huh, looks like the docs gate is broken on nova/crypto.py | |
| 17:58:48 | artom | ^^ somehow failed it | |
| 18:03:10 | artom | Actually, just pep8 in general is broken: https://zuul.opendev.org/t/openstack/build/0b3b9315c9cb4d489f2ffb1355145a99 | |
| 18:04:32 | artom | Ah, it's mypy... | |
| 18:16:14 | lyarwood | artom: the fix for that is in the gate | |
| 18:16:44 | artom | lyarwood, oh. And here I was filing a bug and writing a fix. | |
| 18:16:44 | lyarwood | https://review.opendev.org/c/openstack/nova/+/795533 | |
| 18:17:41 | lyarwood | Always enjoyable when that happens ^_^ | |
| 18:18:23 | artom | I mean, no mailing list message, no bug filed, nothing :P | |
| 18:18:57 | artom | Hrmm, I wonder if my fix could stiff have value | |
| 18:21:26 | artom | I guess not, since we'll need to manually update requirements anyways? | |
| 18:21:54 | artom | I did it by just adding `python -m mypy --install-types` to tools/mypywrap.sh | |
| 18:23:31 | lyarwood | Yeah I think stephenfin tried that earlier | |
| 18:23:40 | lyarwood | but it wasn't enough in the gate for some reason | |
| 18:24:12 | artom | I guess because lower-constraints.txt? | |
| 18:42:59 | cz3 | This is probably a trivial question... I wanted to import Nova modules into interactive Python shell, but I cannot figure out how to load the config from default nova.conf. Is `CONF = nova.conf.CONF` supposed to handle this? | |
| 18:43:54 | cz3 | I wanted to play around with objects module through interactive shell but the lack of loaded config is blocking me | |
| 18:45:45 | artom | cz3, never tried it myself, but looking at the code, yeah, import nova.conf; CONF = nova.conf.CONF should do it | |
| 18:49:26 | cz3 | yeah, that's what I tried... doesn't seem to work. looks like it is still trying to use the defaults. I assume that because it tries to look up things in sqlite instead of mysql that I have in nova.conf | |
| 18:51:00 | artom | Maybe it's not finding the config file, and using the defaults... | |
| 18:52:30 | artom | In fact, if you're importing it via Python (and not via running a Python executable on the command line), I'm not even sure how it finds the config file to use... | |
| 18:54:21 | artom | I *think* you want to do CONF = nova.conf.CONF(project='nova') in such a case... | |
| 18:54:29 | artom | And then it'll look for nova.conf in the "normal" places | |
| 18:56:06 | cz3 | yes, I will poke around the nova.conf.CONF, maybe it needs some additional parameters when it is imported into interactive shell | |
| 18:56:29 | sean-k-mooney | cz3: you need to initalise the conf form the file too | |
| 18:56:40 | sean-k-mooney | if you just import it it will get the default form code | |
| 18:57:42 | sean-k-mooney | we do this https://github.com/openstack/nova/blob/master/nova/config.py#L94-L98 | |
| 18:57:53 | sean-k-mooney | where conf is CONF = nova.conf.CONF | |
| 19:02:02 | sean-k-mooney | all the command lets initalise it here https://github.com/openstack/nova/blob/d64edd3da2336a5c7c8f69cced45272cbaf638a9/nova/cmd/status.py#L374 | |
| 19:02:08 | sean-k-mooney | well liek ^ | |
| 19:03:24 | sean-k-mooney | so "import nova.conf; from nova import config; CONF = nova.conf.CONF; config.parse_args(sys.argv)" | |
| 19:03:41 | sean-k-mooney | cz3: if you do that i think it should work | |
| 19:04:59 | sean-k-mooney | although im not sure you need to initalise rpc or configre the db | |
| 19:05:21 | cz3 | in this case I just need the db access | |
| 19:06:49 | sean-k-mooney | which looks like its jsut sqlalchemy_api.configure(CONF) | |
| 19:07:26 | cz3 | sean-k-mooney: config.parse_args did the trick | |
| 19:07:31 | cz3 | thank you! | |
| 19:07:31 | sean-k-mooney | in any case i would use https://github.com/openstack/nova/blob/master/nova/cmd/manage.py as a reference | |
| 19:07:43 | sean-k-mooney | no worries | |
| 19:08:03 | sean-k-mooney | CONF(argv[1:], | |
| 19:08:05 | sean-k-mooney | project='nova', | |
| 19:08:07 | sean-k-mooney | version=version.version_string(), | |
| 19:08:09 | sean-k-mooney | default_config_files=default_config_files) | |
| 19:08:11 | sean-k-mooney | that is likely the bit you really needed | |
| 19:08:37 | sean-k-mooney | that is invokeing the gloabl constuctor for the singelton | |
| 19:09:11 | cz3 | I actually aim to add something to nova-manage and send it over for review, just wanted to test few things beforehand | |
| 19:09:35 | cz3 | I emailed the openstack-discuss list last week about this, it was re: request_specs regeneration | |
| 19:09:56 | cz3 | where I wanted to find a way to sync extra_specs from flavor data into existing instances | |
| 19:10:33 | sean-k-mooney | oh yes i remember | |
| 19:10:39 | cz3 | long time ago I did something like this by directly appending stuff into request_spec json in database, but this time I want to do it the proper way | |
| 19:13:20 | cz3 | I know that flavors are supposed to be immutable etc. but in this case I just want the existing request_specs to reflect the actual reality and not make the scheduler migrate instances where it was not supposed to after underlying flavor was updated with extra_specs | |
| 19:15:05 | cz3 | I will try to add that to nova-manage and send it over to review via gerrit, maybe it will be good enough to get upstreamed | |
| 19:15:54 | sean-k-mooney | the main consurn we woudl have it that such an update can lead to the vm being broken or on an invalid host also | |
| 19:16:30 | sean-k-mooney | on one hand nova manage is admin only and require direct acess to the server hosting the cloud to get the configs | |
| 19:16:59 | sean-k-mooney | but from a downstream product point of view im not sure we would support customer executing this command | |
| 19:17:16 | sean-k-mooney | we may be with the caveat tha that if you break things you fix them | |
| 19:17:33 | sean-k-mooney | its something we would have to think about | |
| 19:18:01 | cz3 | yeah, I do share the concern, I don't want to send something that would end up breaking deployments for other people if used improperly | |
| 19:18:35 | sean-k-mooney | you proably would need to run som of the flavor/image compatiabity checks before doing the update to make it safe | |
| 19:18:53 | sean-k-mooney | our add a --yes-i-really-really-mean-it flag | |
| 19:19:14 | sean-k-mooney | anyway o/ time to go have dinner | |
| 19:19:42 | cz3 | sure | |
| 19:20:05 | cz3 | thanks again! | |
| 21:57:38 | opendevreview | melanie witt proposed openstack/nova stable/ussuri: zuul: Start to migrate nova-live-migration to zuulv3 https://review.opendev.org/c/openstack/nova/+/795432 | |
| 21:57:39 | opendevreview | melanie witt proposed openstack/nova stable/ussuri: Remove broken legacy zuul jobs https://review.opendev.org/c/openstack/nova/+/795374 | |
| 22:07:55 | tosky | melwitt: an supersquash! Would it make sense to change the first line of the commit message? It is more than just that single commit it was taken from | |
| 22:21:29 | opendevreview | melanie witt proposed openstack/nova master: api: Log correct client IP if load balancer in use https://review.opendev.org/c/openstack/nova/+/786766 | |
| 22:25:02 | gmann | stephenfin: can you check if anything else you want to add in this spec, leaving +w to you https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| #openstack-nova - 2021-06-10 | |||
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Add functional regression test for bug 1853009 https://review.opendev.org/c/openstack/nova/+/695012 | |