Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-17
16:02:33 shilpasd mriedem: efried: understood
16:02:56 mriedem sean-k-mooney: consider that in rocky (i think it was) we fixed scheduling and the resource tracker to not request disk for volume-backed servers,
16:03:02 sean-k-mooney people keep telling me there is a reason we allow bfv instance to have a root_gb!=0
16:03:10 mriedem which would indirectly cause the os-hypervisors API to no longer report disk usage for those volume-backed servers,
16:03:19 mriedem so yes it changes the api behavior, but it's fixing latent busted behavior
16:03:29 sean-k-mooney sure
16:03:54 sean-k-mooney but im wonder ing why we allow bfv guests to have a root_gb set to non 0 in general
16:04:12 mriedem we don't by default policy anymore
16:04:15 sean-k-mooney we dont allow non bfv flavor with root_gb=0
16:04:31 mriedem see os_compute_api:servers:create:zero_disk_flavor
16:04:32 sean-k-mooney i think that is only in the ohter direction
16:04:40 mriedem https://docs.openstack.org/nova/latest/configuration/policy.html
16:05:18 mriedem and sqla-migrate doesn't use upper-constraints because...it's never needed to
16:05:20 sean-k-mooney ya that is not what im asking
16:05:27 mriedem it hardly changes, is released like once per year
16:05:28 sean-k-mooney mriedem: ok
16:05:43 sean-k-mooney ill remove the todo
16:05:52 sean-k-mooney the one i was adding
16:06:01 mriedem sean-k-mooney: also https://review.opendev.org/#/c/669738/
16:06:19 sean-k-mooney so that policy flag prevent instace with root disk = 0 if they are not bfv
16:06:29 openstackgerrit Balazs Gibizer proposed openstack/nova master: Pull up compute node queries to init_host https://review.opendev.org/682680
16:06:29 mriedem correct
16:06:41 shilpasd mriedem: what is sqla-migrate?
16:06:54 sean-k-mooney but we dont have a simplar policy for the oppisite
16:07:07 mriedem sqlalchemy-migrate
16:07:15 sean-k-mooney e.g. prevent bfv instance using a flavor with root_gb!=0
16:07:19 shilpasd ohh, tnx, got it
16:07:32 sean-k-mooney that is the policy i think we should consider adding
16:07:55 mriedem so you want to force all deployers to duplicate all of their flavors?
16:08:05 sean-k-mooney the main objection i have heard to that is requireing two sets of flavor. 1 for bfv and 1 for non bfv
16:08:18 mriedem i would not do that without some additional feature to make bfv simpler with volume-defined flavors or something
16:08:44 mriedem https://review.opendev.org/#/c/511965/
16:09:01 mriedem let's just do that in V
16:09:09 mriedem V for Volumessssss
16:09:24 sean-k-mooney well it woudl be a policy flag taht coudl default to false so not force them but im not sure how common it is to mix bfv and non bfv on the same host
16:09:57 sean-k-mooney i got the impression people tend to segration the bfv hosts form the rest but i could be wrong
16:11:10 shilpasd mriedem: sean-k-mooney: it will be great help, if you add review comments on patch so that can be addressed appropriately https://review.opendev.org/#/c/612626/
16:12:25 sean-k-mooney mriedem_afk: oh thanks for reviewig the config drive stuff ill fix that up
16:14:21 openstackgerrit Merged openstack/nova master: Remove SchedulerReportClient from AggregateRequestFiltersTest https://review.opendev.org/682480
16:14:29 openstackgerrit Merged openstack/nova stable/stein: Remove descriptions of nonexistent hacking rules https://review.opendev.org/679695
16:21:02 sean-k-mooney efried: stephenfin do either of ye have the rc bug etherpad link
16:21:17 efried https://etherpad.openstack.org/p/nova-train-release-todo ?
16:21:25 sean-k-mooney yes that one
16:21:59 sean-k-mooney efried: https://review.opendev.org/#/c/669738/4 would be a regression in train if we dont land it in the rc
16:22:55 efried sean-k-mooney: ack
16:23:00 efried make sure mriedem_afk is aware
16:23:09 sean-k-mooney efried: he is
16:23:18 sean-k-mooney im going to address his review feedback now
16:23:33 sean-k-mooney we "fixed" https://bugs.launchpad.net/nova/+bug/1827492
16:23:34 openstack Launchpad bug 1827492 in OpenStack Compute (nova) stein "vms failed to hard reboot and became error after set force_config_drive in compute nodes" [Low,In progress] - Assigned to Lee Yarwood (lyarwood)
16:23:53 sean-k-mooney and intoduced https://bugs.launchpad.net/nova/+bug/1835822
16:23:54 openstack Launchpad bug 1835822 in OpenStack Compute (nova) "vms loose acess to config drive with CONF.force_config_drive=True after hard reboot" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney)
16:43:53 zzzeek efried: how does that huge amount of change in the .zuul.yaml file relate to the extra config variable being set ?
16:44:33 efried zzzeek: It just restricts the run to only do unit tests, and gather logs from the mysql dir
16:44:47 efried zzzeek: if we end up doing this irl we'll revert that bit and just keep the test setup bit.
16:45:05 zzzeek efried: oh ok
16:45:16 zzzeek efried: well great, see if it fixes those errors
16:45:26 efried The extra zuul job (*collect) is to actually grab the logs so we can make sure this works. The removals is just so we don't waste the gate resources on the other test jobs.
16:45:41 efried zzzeek: thanks for the pointer!
16:51:37 sean-k-mooney efried: i just commented on the mail but could the mysql issues be related too the the rabbit mq issue we have in the api
16:52:01 sean-k-mooney e.g. could it be cause by eventlet and running multiple thread in the wsgi server
16:52:16 sean-k-mooney im not sure if this is happeng in the gate or not
16:52:21 sean-k-mooney if its in the gate then no
16:52:23 efried sean-k-mooney: I have no. freaking. clue. If you can solve this one, I will buy you cookies forever.
16:52:42 efried it's happening in the gate, yeah.
16:52:54 sean-k-mooney ok in the gate we hard code one thread i think
16:52:57 sean-k-mooney but ill check
16:52:59 efried wait
16:53:04 efried we're talking about different issues, sorry.
16:53:16 efried disregard cookie promise
16:53:25 sean-k-mooney is was talking about RE: [oslo][nova] Nova causes MySQL timeouts
16:53:38 efried cookie promise applies to http://bugs.launchpad.net/bugs/1823251
16:53:39 openstack Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed]
16:53:41 sean-k-mooney which i assme is what you and zzzeek were talking about?
16:53:51 efried no, zzzeek and I were talking about ----^
16:53:51 sean-k-mooney ah ok
17:16:11 efried sean-k-mooney: oh, it looks like it may actually be the same thing zzzeek and I were talking about.
17:16:18 efried which kind of forked off of the test_walk_versions thing.
17:17:02 sean-k-mooney the mysql timeouts
17:17:28 sean-k-mooney so i was wondering if it could be related to eventlet in genreal
17:18:28 sean-k-mooney connection pooling might not play nicely with evnetly monkey patching when don in a wsgi envionment
17:18:49 sean-k-mooney or rather a mod_wsgi or uwsgi env
17:19:36 sean-k-mooney that said if it was an eventlet issue i dont think it would affect the test_walk_versions stuff
17:21:09 efried sean-k-mooney: it may be entirely unrelated to the gate timeout in those tests. But I'm seeing it in the mysqld logs for those tests (whether they pass or fail). So, can't hurt to get rid of them, nah?
17:23:56 sean-k-mooney well i thik fixing mysql issue in general is a good thing
17:24:12 sean-k-mooney just not sure as you said fi they are releated or not
17:25:29 openstackgerrit François Palin proposed openstack/nova master: Add retry when attachment_delete or terminate_connection fails https://review.opendev.org/669674
17:51:23 openstackgerrit sean mooney proposed openstack/nova master: make config drives sticky bug 1835822 https://review.opendev.org/669738
17:51:24 openstack bug 1835822 in OpenStack Compute (nova) "vms loose acess to config drive with CONF.force_config_drive=True after hard reboot" [Medium,In progress] https://launchpad.net/bugs/1835822 - Assigned to sean mooney (sean-k-mooney)
17:52:39 mriedem_afk sean-k-mooney: i tagged the bug so we won't forget about it https://bugs.launchpad.net/nova/+bugs?field.tag=train-rc-potential
17:52:54 donnyd I fixed my mysql issues by removing the ulimits in systemd for the service
17:52:56 sean-k-mooney mriedem_afk: i added it to the etherpad
17:53:12 donnyd probably a better answer out there... but i haven't ran into it again
17:53:17 sean-k-mooney mriedem_afk: i think i have done the things you asked for
17:54:46 sean-k-mooney ah i see you are using launchpad to track the rc potentail bugs ya that is better
17:55:12 sean-k-mooney im going to grab food but ill asress your comments in the devname patch when i get back
18:05:13 mriedem sean-k-mooney: https://review.opendev.org/#/c/669738/5
18:37:47 efried donnyd: Were they the same mysql issues we've been discussing?
18:38:25 efried I sort of doubt it would be a problem to remove ulimits on CI nodes?
18:47:06 donnyd yea it was pretty close to the same issue

Earlier   Later