| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-09 | |||
| 14:50:55 | gibi | sean-k-mooney: please put it up to the agenda. Honestly I don't think it worth the effort to do extra things with placement but it make sense to discuss it | |
| 14:52:12 | sean-k-mooney | oh i was suggesting adding a "check for open placment bugs" item | |
| 14:52:19 | sean-k-mooney | like when we do the nova bug review | |
| 14:52:38 | sean-k-mooney | more just to prompt use to look for placment bugs if tehy areever reported | |
| 14:53:00 | sean-k-mooney | most get brought up here anyway since nova typicly is what finds them if they exist | |
| 14:57:13 | gibi | honestly I don't belive that people will start looking at open bug reports in placement by that prompt. but I might be negative | |
| 15:01:31 | sean-k-mooney | ok then lets leave it be so | |
| 15:03:43 | melwitt | tosky: guh, ok, I was wondering if there might be something like that since the fails. I'll look into a potential squash, thanks for the hint | |
| 15:04:00 | gibi | I just don't have the bandwidth to do a proper care for placement as well, but it seem that placement does not require that level of care most of the time | |
| 15:05:02 | masterpe[m] | Hi, we have Train running, yesterday we got the error "No valid host was found. There are not enough hosts available" on our platform. If I count the free RAM and vcpu of the nodes, I should have enough resources left to get the instances started. Is there something that I miss? | |
| 15:05:36 | melwitt | sean-k-mooney, gibi: I knew that placement had its own specs repo but I thought it was intentional that proposed spec to the nova-specs repo since the projects are back together again /shrug | |
| 15:09:53 | melwitt | gibi: there is a review I have hanging around if you might be able to spare some time, it is not urgent https://review.opendev.org/c/openstack/nova/+/780395 | |
| 15:15:38 | sean-k-mooney | melwitt: ya so i didnt know i we had decied to not use the in repos specs or not | |
| 15:15:54 | sean-k-mooney | e.g. if we had jsut defaulted to nova-spec or if it was intentional | |
| 15:16:31 | melwitt | sean-k-mooney: no, I'm not saying there was a decision, I had assumed something like that had occurred because of where the re-parent spec was proposed, but I was wrong | |
| 15:19:52 | gibi | melwitt: queued that up for review | |
| 15:20:24 | melwitt | thank you gibi | |
| 15:26:23 | opendevreview | Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575 | |
| 15:26:24 | melwitt | masterpe[m]: you need to look at the debug level logs in the placement api logs and the nova-scheduler logs to see details about why no valid host was found | |
| 15:28:39 | masterpe[m] | To solve it for now I enabled a node that was disabled/marked as spare. So I need to enable debug mode to troubleshoot? | |
| 15:32:42 | melwitt | masterpe[m]: to get full detail, yes. what you can do without debug mode is check whether there is a message like "no allocation candidates" in the nova-scheduler log, if you see that, it's likely cpu/ram/disk (as those resources are filtered by placement service) | |
| 15:33:42 | melwitt | if you don't see that, it means one or more of the nova side scheduler filters eliminated all hosts | |
| 15:34:11 | opendevreview | Balazs Gibizer proposed openstack/placement master: [DNM]: test consumer_type rollback due to consumer_gen conflict https://review.opendev.org/c/openstack/placement/+/795577 | |
| 15:36:31 | melwitt | gibi: oh heh, I had wondered if you saw my latest confusing comment. I was thinking the same thing this morning that it would be best to check what happens with a test. thanks for doing that | |
| 15:36:59 | gibi | melwitt: yeah, you are right the whole PUT /allocations/{uuid} call is not a single transaction | |
| 15:37:28 | gibi | so with the current implementation we can leak a consumer_type update from a rejected call :/ | |
| 15:39:31 | gibi | unfortunatly create a test case for a real race is too hard | |
| 15:39:43 | melwitt | gibi: ah, I see your new comment on the review. yeah ok, agree we need something higher up the stack then in order to not leak (or some other better idea yet to be thought of) :( I will work on it | |
| 15:40:32 | gibi | it is hard :/ | |
| 15:42:25 | melwitt | yeah, would require faking things with mocks at best. I didn't explore that yet so not sure how messy or possible | |
| 15:46:49 | spatel | sean-k-mooney periodically i am seeing this issue on my nova logs not sure if this is bug or not - http://paste.openstack.org/show/806496/ | |
| 15:51:45 | melwitt | spatel: that error means connection to the database failed while trying to write a service record update (heartbeat), not a bug. error being emitted from pymysql client | |
| 15:52:41 | spatel | melwitt today i said terraform to build 20 vm and 4 vm failed and stick in BUILD process, when i check their compute nodes i found that error | |
| 15:55:44 | melwitt | spatel: that paste wouldn't be directly related to a build failure (that paste is a service record update for a service heartbeat) so you'd need to look at which errors are related to the build fail, if any | |
| 16:03:13 | spatel | let me see, may be issue of rabbitmq in that case | |
| 16:05:41 | melwitt | dansmith: someone linked this review in the freenode channel and I thought you might have opinion about it. it's a proposal to allow config of rbd direct snapshot to enable an operator to disallow the fast clone, with the use case being the operator is migrating the backend and the fast clone wouldn't honor the glance backend config (I don't fully understand this part) https://review.opendev.org/c/openstack/nova/+/794837 | |
| 16:30:53 | spatel | melwitt also noticed periodically rabbitMQ connection reset error from nova-api - http://paste.openstack.org/show/806498/ | |
| 16:31:06 | spatel | is this a normal behavior ? | |
| 16:35:44 | melwitt | spatel: it's not uncommon but it's not desirable. I have seen it many times, related to environmental issues like network disruptions. it's also similar to the DBConnectionError you pasted earlier, that one was about the pymysql client failing to be able to connect to the database, these rabbit errors are showing connection reset or closed trying to connect to rabbitmq | |
| 16:37:05 | spatel | These controller and compute node on same rack | |
| 16:37:19 | spatel | let me restart my whole rabbitMQ cluster | |
| 16:38:32 | spatel | when i delete vm they getting stuck in deleting state | |
| 17:14:36 | dansmith | melwitt: ack, thanks | |
| 17:39:55 | opendevreview | Artom Lifshitz proposed openstack/nova master: docs: Explicitly state lack of support for node renaming https://review.opendev.org/c/openstack/nova/+/795607 | |
| 17:58:43 | artom | Huh, looks like the docs gate is broken on nova/crypto.py | |
| 17:58:48 | artom | ^^ somehow failed it | |
| 18:03:10 | artom | Actually, just pep8 in general is broken: https://zuul.opendev.org/t/openstack/build/0b3b9315c9cb4d489f2ffb1355145a99 | |
| 18:04:32 | artom | Ah, it's mypy... | |
| 18:16:14 | lyarwood | artom: the fix for that is in the gate | |
| 18:16:44 | artom | lyarwood, oh. And here I was filing a bug and writing a fix. | |
| 18:16:44 | lyarwood | https://review.opendev.org/c/openstack/nova/+/795533 | |
| 18:17:41 | lyarwood | Always enjoyable when that happens ^_^ | |
| 18:18:23 | artom | I mean, no mailing list message, no bug filed, nothing :P | |
| 18:18:57 | artom | Hrmm, I wonder if my fix could stiff have value | |
| 18:21:26 | artom | I guess not, since we'll need to manually update requirements anyways? | |
| 18:21:54 | artom | I did it by just adding `python -m mypy --install-types` to tools/mypywrap.sh | |
| 18:23:31 | lyarwood | Yeah I think stephenfin tried that earlier | |
| 18:23:40 | lyarwood | but it wasn't enough in the gate for some reason | |
| 18:24:12 | artom | I guess because lower-constraints.txt? | |
| 18:42:59 | cz3 | This is probably a trivial question... I wanted to import Nova modules into interactive Python shell, but I cannot figure out how to load the config from default nova.conf. Is `CONF = nova.conf.CONF` supposed to handle this? | |
| 18:43:54 | cz3 | I wanted to play around with objects module through interactive shell but the lack of loaded config is blocking me | |
| 18:45:45 | artom | cz3, never tried it myself, but looking at the code, yeah, import nova.conf; CONF = nova.conf.CONF should do it | |
| 18:49:26 | cz3 | yeah, that's what I tried... doesn't seem to work. looks like it is still trying to use the defaults. I assume that because it tries to look up things in sqlite instead of mysql that I have in nova.conf | |
| 18:51:00 | artom | Maybe it's not finding the config file, and using the defaults... | |
| 18:52:30 | artom | In fact, if you're importing it via Python (and not via running a Python executable on the command line), I'm not even sure how it finds the config file to use... | |
| 18:54:21 | artom | I *think* you want to do CONF = nova.conf.CONF(project='nova') in such a case... | |
| 18:54:29 | artom | And then it'll look for nova.conf in the "normal" places | |
| 18:56:06 | cz3 | yes, I will poke around the nova.conf.CONF, maybe it needs some additional parameters when it is imported into interactive shell | |
| 18:56:29 | sean-k-mooney | cz3: you need to initalise the conf form the file too | |
| 18:56:40 | sean-k-mooney | if you just import it it will get the default form code | |
| 18:57:42 | sean-k-mooney | we do this https://github.com/openstack/nova/blob/master/nova/config.py#L94-L98 | |
| 18:57:53 | sean-k-mooney | where conf is CONF = nova.conf.CONF | |
| 19:02:02 | sean-k-mooney | all the command lets initalise it here https://github.com/openstack/nova/blob/d64edd3da2336a5c7c8f69cced45272cbaf638a9/nova/cmd/status.py#L374 | |
| 19:02:08 | sean-k-mooney | well liek ^ | |
| 19:03:24 | sean-k-mooney | so "import nova.conf; from nova import config; CONF = nova.conf.CONF; config.parse_args(sys.argv)" | |
| 19:03:41 | sean-k-mooney | cz3: if you do that i think it should work | |
| 19:04:59 | sean-k-mooney | although im not sure you need to initalise rpc or configre the db | |
| 19:05:21 | cz3 | in this case I just need the db access | |
| 19:06:49 | sean-k-mooney | which looks like its jsut sqlalchemy_api.configure(CONF) | |
| 19:07:26 | cz3 | sean-k-mooney: config.parse_args did the trick | |
| 19:07:31 | cz3 | thank you! | |
| 19:07:31 | sean-k-mooney | in any case i would use https://github.com/openstack/nova/blob/master/nova/cmd/manage.py as a reference | |
| 19:07:43 | sean-k-mooney | no worries | |
| 19:08:03 | sean-k-mooney | CONF(argv[1:], | |
| 19:08:05 | sean-k-mooney | project='nova', | |
| 19:08:07 | sean-k-mooney | version=version.version_string(), | |
| 19:08:09 | sean-k-mooney | default_config_files=default_config_files) | |
| 19:08:11 | sean-k-mooney | that is likely the bit you really needed | |
| 19:08:37 | sean-k-mooney | that is invokeing the gloabl constuctor for the singelton | |
| 19:09:11 | cz3 | I actually aim to add something to nova-manage and send it over for review, just wanted to test few things beforehand | |
| 19:09:35 | cz3 | I emailed the openstack-discuss list last week about this, it was re: request_specs regeneration | |
| 19:09:56 | cz3 | where I wanted to find a way to sync extra_specs from flavor data into existing instances | |
| 19:10:33 | sean-k-mooney | oh yes i remember | |
| 19:10:39 | cz3 | long time ago I did something like this by directly appending stuff into request_spec json in database, but this time I want to do it the proper way | |
| 19:13:20 | cz3 | I know that flavors are supposed to be immutable etc. but in this case I just want the existing request_specs to reflect the actual reality and not make the scheduler migrate instances where it was not supposed to after underlying flavor was updated with extra_specs | |
| 19:15:05 | cz3 | I will try to add that to nova-manage and send it over to review via gerrit, maybe it will be good enough to get upstreamed | |
| 19:15:54 | sean-k-mooney | the main consurn we woudl have it that such an update can lead to the vm being broken or on an invalid host also | |
| 19:16:30 | sean-k-mooney | on one hand nova manage is admin only and require direct acess to the server hosting the cloud to get the configs | |
| 19:16:59 | sean-k-mooney | but from a downstream product point of view im not sure we would support customer executing this command | |
| 19:17:16 | sean-k-mooney | we may be with the caveat tha that if you break things you fix them | |