| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-11 | |||
| 14:41:20 | dansmith | sean-k-mooney: presumably the host fs has some reserved-for-root set and qemu can't exhaust that though | |
| 14:41:49 | dansmith | sean-k-mooney: agree, overcommit of disk is just asking for problems | |
| 14:41:49 | sean-k-mooney | dansmith: you would hope. | |
| 14:42:11 | sean-k-mooney | peopel get a way with it with lvm thin providiouning by carfully monitoring it | |
| 14:42:24 | sean-k-mooney | but it makes me uncomfortable if i care about the data | |
| 14:42:42 | dansmith | it's the worst thing you can overcommit | |
| 14:43:07 | sean-k-mooney | dansmith: the host_reserved_disk_mb helps but if you allow suspend you can exceed that as we need to snapshot the guest ram and save it to disk | |
| 14:44:12 | sean-k-mooney | when kolla was young it did nto have logrotate and i made the mistake of running our internal intel dev cluserter with debug logging on | |
| 14:44:24 | sean-k-mooney | fixing out of disk was fun... | |
| 14:44:40 | sean-k-mooney | because it caused db curruption | |
| 14:44:55 | dansmith | I think stephenfin gets the picture.. "it's bad, yo" | |
| 14:45:35 | sean-k-mooney | stephenfin: are you going to write doc to cover the interaction betweeen file backed memory and reserved disk by the way | |
| 14:46:05 | sean-k-mooney | if not i can proably try to write something | |
| 14:46:17 | stephenfin | you mean above what I put in the config opt docs here https://review.opendev.org/#/c/734658/ ? | |
| 14:46:43 | stephenfin | oh, reserved _disk_ | |
| 14:46:47 | sean-k-mooney | yes | |
| 14:46:49 | sean-k-mooney | disk | |
| 14:47:12 | sean-k-mooney | and i guess we shoudl menetion the reserved hugepages option too | |
| 14:47:16 | stephenfin | hmm, I hadn't thought of that. Only seeing your comments now | |
| 14:47:32 | stephenfin | Sure, I can do that early next week | |
| 14:48:06 | sean-k-mooney | ok if you dont get to it remind me and i can try to do it | |
| 14:48:36 | sean-k-mooney | basically if its the same file system they shoudl add the value of the file backed memory option to reserved disk | |
| 15:15:36 | aarents | stephenfin: if host disk is full qemu may PAUSE himself (by default I think) see man qemu werror=action,rerror=action) | |
| 15:16:34 | sean-k-mooney | dansmith: looks like glance started listening on http https://zuul.opendev.org/t/openstack/build/8afb850f7844450389823d7f84b90c61/log/controller/logs/screen-g-api.txt#345 but devstack seams to be checking https https://zuul.opendev.org/t/openstack/build/8afb850f7844450389823d7f84b90c61/log/controller/logs/devstacklog.txt#19995 | |
| 15:16:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add a guide on overcommit https://review.opendev.org/735175 | |
| 15:16:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tox: Stop linting as part of docs target https://review.opendev.org/735176 | |
| 15:17:04 | dansmith | that wsgi thing works for me locally | |
| 15:17:11 | stephenfin | sean-k-mooney, dansmith, aarents: that's the doc anyway | |
| 15:17:33 | sean-k-mooney | dansmith: i can deploy with the local.conf that was generated and see what happens | |
| 15:17:39 | dansmith | sean-k-mooney: it's getting a 503, which means it's hitting apache or something which should be proxying | |
| 15:17:56 | sean-k-mooney | im wondering if it enabling the tls code | |
| 15:18:10 | sean-k-mooney | e.g. the tls proxy | |
| 15:18:20 | sean-k-mooney | but not actully configuring it for some reason | |
| 15:18:33 | dansmith | idk | |
| 15:18:51 | sean-k-mooney | the tls proxy is enabled | |
| 15:18:55 | sean-k-mooney | https://94b38a054383d32ffbe4-eecb7ff79b5c6e121d6c20fabad8af79.ssl.cf1.rackcdn.com/734184/10/check/nova-ceph-multistore/8afb850/controller/logs/local_conf.txt | |
| 15:19:03 | dansmith | unless something changed recently | |
| 15:19:29 | dansmith | maybe the ceph plugin job I'm inheriting from does that? | |
| 15:19:42 | dansmith | because at least a couple weeks ago, devstack didn't with just that wsgi switch in the localrc | |
| 15:19:49 | sean-k-mooney | dansmith: i think its enabeld by default | |
| 15:20:09 | dansmith | in d-g? because it wasn't for me, unless it's changed very recently | |
| 15:20:11 | sean-k-mooney | in the devstack or tempest job | |
| 15:20:36 | sean-k-mooney | not d-g but the base devstack zuulv3 job i think | |
| 15:20:42 | dansmith | yeah, that's what I mean | |
| 15:20:45 | dansmith | something other than devstack defaults | |
| 15:21:00 | dansmith | well, I'd not think this would change it, unless the base devstack job breaks with wsgi | |
| 15:21:12 | dansmith | maybe glance has a mod_wsgi job I can look at | |
| 15:21:28 | dansmith | doesn't look like it | |
| 15:22:22 | sean-k-mooney | which raiase the question how do they test this? | |
| 15:22:45 | aarents | sean-k-mooney: dansmith Should I need a reno for a new config parameter ? | |
| 15:22:49 | sean-k-mooney | looking at there zuul jobs they only use the normal tepest jobs | |
| 15:22:51 | dansmith | 2020-06-11 14:59:39.532624 [proxy:error] [pid 15204:tid 140023944947456] (111)Connection refused: AH00957: HTTP: attempt to connect to 10.0.1.13:19292 (10.0.1.13) failed | |
| 15:23:04 | dansmith | sean-k-mooney: yeah .... | |
| 15:23:15 | sean-k-mooney | dansmith: am its runing on a different port | |
| 15:23:47 | dansmith | yep | |
| 15:23:57 | sean-k-mooney | wsgi starting up on http://0.0.0.0:19191 | |
| 15:24:05 | sean-k-mooney | so close | |
| 15:24:15 | dansmith | GLANCE_URL is different for uwsgi | |
| 15:24:19 | dansmith | so maybe I need to set that | |
| 15:24:44 | sean-k-mooney | proably arent you having fun working on glance things... | |
| 15:24:47 | dansmith | or just change the service port | |
| 15:24:51 | dansmith | no :) | |
| 15:25:34 | dansmith | hmm, | |
| 15:25:41 | dansmith | 19191 is the registry port supposedly | |
| 15:25:52 | dansmith | and 19292 is the glance_service_port | |
| 15:26:55 | sangeet | I am trying to get some help on turning on TLS for nova. I was able to but nova-api is not interpreting the messages correctly. I had posed this opensatck channel as well but now response. Any help will be much apprecaited. Nova.conf is http://paste.openstack.org/show/794645 | |
| 15:26:58 | sangeet | 192.168.243.242 - - [11/Jun/2020 14:00:22] code 400, message Bad request syntax ('\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03O\xafjh.\xe4\xeb\x88\xfa\x9d\xff\x18\x813\x7f0\xa2\xe0\xb9J`\x15f\xb7=)\xf2}td~\x0e\x00\x00n\xc00\xc0,\xc02\xc0.\xc0/\xc0+\xc01\xc0-\x00\xa5\x00\xa3\x00\xa1\x00\x9f\x00\xa4\x00\xa2\x00\xa0\x00\x9e\xc0(\xc0$\xc0\x14\xc0') | |
| 15:27:33 | sangeet | Above log is from nova-api | |
| 15:28:04 | sean-k-mooney | sangeet: so nova itslef does not support tls you have to use a webserver to terminate the tls session before it hits the nova api | |
| 15:29:16 | sean-k-mooney | that kind of looks like the api is getting the encyped payload | |
| 15:45:23 | sangeet | Thank you sean-k-mooney | |
| 15:49:46 | gibi | nova meeting starts in 10 minutes on #openstack-meeting-3 | |
| 16:00:00 | jsuchome | dansmith: hi, thanks for your latest review, do you want me to reduce that comment about cyclic import or is there anything else? | |
| 16:05:12 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Try to make a glance multistore job https://review.opendev.org/734184 | |
| 16:13:46 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Update the testing section of the rbd-glance-multistore spec https://review.opendev.org/735192 | |
| 16:14:42 | dansmith | jsuchome: personally I think it should be reduced in line with the other examples I quoted | |
| 16:15:01 | dansmith | jsuchome: I don't remember if there was anything else, but I'll have to circle back next week.. trying to get some stuff done before I disappear tomorrow | |
| 16:45:51 | sean-k-mooney | dansmith: this is the cyborg shelve path fyi https://review.opendev.org/#/c/729563/2 | |
| 16:46:26 | bauzas | gibi: fwiw, I'd have loved to volunteer for helping you during your holidays, but I'll be off at the same time | |
| 16:46:59 | gibi | bauzas: ack, thanks | |
| 16:47:10 | gibi | and don't worry | |
| 16:57:45 | dansmith | sean-k-mooney: ack, will try to keep that in mind in the coming weeks | |
| 17:01:00 | sean-k-mooney | i would like to see ci test it first so ill update this https://review.opendev.org/#/c/709641/ | |
| 17:08:53 | gibi | 19:08 -ChanServ(ChanServ@services.)- You are not authorized to (de)op gibi on | |
| 17:08:56 | gibi | #openstack-nova. | |
| 17:09:12 | gibi | so I need help to update the challen topic to point to the victori runway etherpad | |
| 17:30:43 | sean-k-mooney | gibi: i think dansmith has op on this channel? | |
| 17:31:38 | sean-k-mooney | hehe | |
| 17:31:47 | dansmith | good? | |
| 17:32:04 | sean-k-mooney | seams to work for me | |
| 17:32:22 | sean-k-mooney | and /topic works so yep | |
| 17:34:15 | gibi | dansmith: thanks | |
| 17:44:50 | openstackgerrit | Dan Smith proposed openstack/nova master: Check cherry-pick hashes in pep8 tox target https://review.opendev.org/735205 | |
| 17:45:31 | dansmith | sean-k-mooney: did you have a stable commit with broken hashes in gerrit that is still yet to be fixed? | |
| 17:45:34 | dansmith | would like to use it to test | |
| 18:04:39 | dansmith | sean-k-mooney: ^ ? | |
| 18:05:06 | sean-k-mooney | am im not sure if stephen fixed it | |
| 18:05:15 | sean-k-mooney | ill check | |
| 18:05:56 | dansmith | okay I wasn't sure if you had proposed it or someone else.. I don't see any recent -1ed patches on stable from you | |
| 18:06:03 | sean-k-mooney | https://review.opendev.org/#/c/728057/ was the one i think | |